19/08/07 12:40:15 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead. 19/08/07 12:40:15 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead. 19/08/07 12:40:15 INFO AccessManager: AccessControlManager acls enabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: * 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Welcome to 19/08/07 12:40:16 INFO LineBufferedStream: stdout: ____ __ 19/08/07 12:40:16 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__ 19/08/07 12:40:16 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/ 19/08/07 12:40:16 INFO LineBufferedStream: stdout: /___/ .__/\_,_/_/ /_/\_\ version 2.3.2.3.1.0.0-78 19/08/07 12:40:16 INFO LineBufferedStream: stdout: /_/ 19/08/07 12:40:16 INFO LineBufferedStream: stdout: 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_112 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Branch HEAD 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2018-12-06T12:26:34Z 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Revision 9b78096afddf26e2d73f0c078a112c9bf979ed53 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git 19/08/07 12:40:16 INFO LineBufferedStream: stdout: Type --help for more information. 19/08/07 12:40:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/08/07 12:40:22 INFO RMProxy: Connecting to ResourceManager at bda01.vm.altran/10.0.0.157:8050 19/08/07 12:40:28 INFO AHSProxy: Connecting to Application History server at bda02.vm.altran/10.0.0.156:10200 19/08/07 12:40:28 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 19/08/07 12:40:28 INFO StateStore$: Using FileSystemStateStore for recovery. 19/08/07 12:40:28 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0 19/08/07 12:40:28 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0 19/08/07 12:40:28 INFO InteractiveSessionManager: Heartbeat watchdog thread started. 19/08/07 12:40:28 INFO LivyServer: CSRF protection is enabled. 19/08/07 12:40:28 INFO LivyServer: Access control is enabled 19/08/07 12:40:29 INFO WebServer: Starting server on http://bda01.vm.altran:8999 19/08/08 11:38:35 INFO InteractiveSession$: Creating Interactive session 0: [owner: null, request: [kind: shared, proxyUser: None, conf: spark.driver.memory -> 1g,spark.executor.instances -> 1,spark.driver.cores -> 1,livy.uri -> http://10.0.0.157:8999,spark.executor.memory -> 1g,spark.dynamicAllocation.enabled -> false,spark.executor.cores -> 1, heartbeatTimeoutInSecond: 0]] 19/08/08 11:38:41 INFO RpcServer: Connected to the port 10001 19/08/08 11:38:42 WARN RSCConf: Your hostname, bda01.vm.altran, resolves to a loopback address, but we couldn't find any external IP address! 19/08/08 11:38:42 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. 19/08/08 11:38:45 INFO InteractiveSessionManager: Registering new session 0 19/08/08 11:40:13 INFO LineBufferedStream: stdout: Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. 19/08/08 11:40:13 WARN ContextLauncher: Waiting thread interrupted, killing child process. 19/08/08 11:40:13 ERROR RSCClient: Failed to connect to context. java.util.concurrent.TimeoutException: Timed out waiting for context to start. at org.apache.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:134) at org.apache.livy.rsc.ContextLauncher.access$300(ContextLauncher.java:63) at org.apache.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:122) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:745) 19/08/08 11:40:13 INFO RSCClient: Failing pending job 743c983d-45cc-4a89-9192-11f88295c659 due to shutdown. 19/08/08 11:40:13 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application. 19/08/08 11:40:13 INFO InteractiveSession: Stopping InteractiveSession 0... 19/08/08 11:40:49 INFO InteractiveSession: Stopped InteractiveSession 0. 19/08/08 11:40:49 WARN InteractiveSession: Fail to get rsc uri java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for context to start. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41) at org.apache.livy.server.interactive.InteractiveSession$$anonfun$22.apply(InteractiveSession.scala:393) at org.apache.livy.server.interactive.InteractiveSession$$anonfun$22.apply(InteractiveSession.scala:393) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.util.concurrent.TimeoutException: Timed out waiting for context to start. at org.apache.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:134) at org.apache.livy.rsc.ContextLauncher.access$300(ContextLauncher.java:63) at org.apache.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:122) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:745) 19/08/08 11:40:49 ERROR SparkYarnApp: Error whiling refreshing YARN state java.lang.IllegalStateException: No YARN application is found with tag livy-session-0-h7t5utzg in 120 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details. at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:183) at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:242) at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239) at scala.Option.getOrElse(Option.scala:121) at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:239) at org.apache.livy.Utils$$anon$1.run(Utils.scala:97) 19/08/08 11:49:01 INFO InteractiveSession$: Creating Interactive session 1: [owner: null, request: [kind: shared, proxyUser: None, conf: spark.driver.memory -> 1g,spark.executor.instances -> 1,spark.driver.cores -> 1,livy.uri -> http://10.0.0.157:8999,spark.executor.memory -> 1g,spark.dynamicAllocation.enabled -> false,spark.executor.cores -> 1, heartbeatTimeoutInSecond: 0]] 19/08/08 11:49:01 INFO RpcServer: Connected to the port 10001 19/08/08 11:49:01 WARN RSCConf: Your hostname, bda01.vm.altran, resolves to a loopback address, but we couldn't find any external IP address! 19/08/08 11:49:01 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. 19/08/08 11:49:02 INFO InteractiveSessionManager: Registering new session 1 19/08/08 11:49:02 INFO LineBufferedStream: stdout: Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. 19/08/08 11:49:12 INFO LineBufferedStream: stdout: 19/08/08 11:49:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/08/08 11:49:33 INFO LineBufferedStream: stdout: 19/08/08 11:49:33 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 19/08/08 11:49:33 INFO LineBufferedStream: stdout: 19/08/08 11:49:33 INFO RMProxy: Connecting to ResourceManager at bda01.vm.altran/10.0.0.157:8050 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Requesting a new application from cluster with 1 NodeManagers 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (24576 MB per container) 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Setting up container launch context for our AM 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Setting up the launch environment for our AM container 19/08/08 11:49:41 INFO LineBufferedStream: stdout: 19/08/08 11:49:41 INFO Client: Preparing resources for our AM container 19/08/08 11:49:43 INFO LineBufferedStream: stdout: 19/08/08 11:49:43 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 11:49:43 INFO LineBufferedStream: stdout: 19/08/08 11:49:43 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 11:49:43 INFO LineBufferedStream: stdout: 19/08/08 11:49:43 INFO Client: Distribute hdfs cache file as spark.sql.hive.metastore.jars for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 11:49:43 INFO LineBufferedStream: stdout: 19/08/08 11:49:43 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 11:49:43 INFO LineBufferedStream: stdout: 19/08/08 11:49:43 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-api-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/livy-api-0.5.0.3.1.0.0-78.jar 19/08/08 11:49:45 INFO LineBufferedStream: stdout: 19/08/08 11:49:45 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-rsc-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/livy-rsc-0.5.0.3.1.0.0-78.jar 19/08/08 11:49:46 INFO LineBufferedStream: stdout: 19/08/08 11:49:46 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/netty-all-4.1.17.Final.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/netty-all-4.1.17.Final.jar 19/08/08 11:49:47 INFO LineBufferedStream: stdout: 19/08/08 11:49:47 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/commons-codec-1.9.jar 19/08/08 11:49:48 INFO LineBufferedStream: stdout: 19/08/08 11:49:48 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-core_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/livy-core_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 11:49:49 INFO LineBufferedStream: stdout: 19/08/08 11:49:49 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-repl_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/livy-repl_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 11:49:51 INFO LineBufferedStream: stdout: 19/08/08 11:49:51 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/datanucleus-rdbms-3.2.9.jar 19/08/08 11:49:52 INFO LineBufferedStream: stdout: 19/08/08 11:49:52 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/datanucleus-api-jdo-3.2.6.jar 19/08/08 11:49:53 INFO LineBufferedStream: stdout: 19/08/08 11:49:53 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/datanucleus-core-3.2.10.jar 19/08/08 11:49:54 INFO LineBufferedStream: stdout: 19/08/08 11:49:54 INFO Client: Uploading resource file:/etc/spark2/3.1.0.0-78/0/hive-site.xml -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/hive-site.xml 19/08/08 11:49:56 INFO LineBufferedStream: stdout: 19/08/08 11:49:56 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/R/lib/sparkr.zip#sparkr -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/sparkr.zip 19/08/08 11:49:58 INFO LineBufferedStream: stdout: 19/08/08 11:49:58 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/pyspark.zip 19/08/08 11:49:59 INFO LineBufferedStream: stdout: 19/08/08 11:49:59 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/py4j-0.10.7-src.zip 19/08/08 11:50:00 INFO LineBufferedStream: stdout: 19/08/08 11:50:00 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/pyspark.zip added multiple times to distributed cache 19/08/08 11:50:00 INFO LineBufferedStream: stdout: 19/08/08 11:50:00 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache 19/08/08 11:50:16 INFO LineBufferedStream: stdout: 19/08/08 11:50:16 INFO Client: Uploading resource file:/tmp/spark-5e1fa384-7450-4e34-b150-760102fab560/__spark_conf__5444766395792842078.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0003/__spark_conf__.zip 19/08/08 11:50:17 INFO LineBufferedStream: stdout: 19/08/08 11:50:17 INFO SecurityManager: Changing view acls to: livy 19/08/08 11:50:17 INFO LineBufferedStream: stdout: 19/08/08 11:50:17 INFO SecurityManager: Changing modify acls to: livy 19/08/08 11:50:17 INFO LineBufferedStream: stdout: 19/08/08 11:50:17 INFO SecurityManager: Changing view acls groups to: 19/08/08 11:50:17 INFO LineBufferedStream: stdout: 19/08/08 11:50:17 INFO SecurityManager: Changing modify acls groups to: 19/08/08 11:50:17 INFO LineBufferedStream: stdout: 19/08/08 11:50:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy); groups with view permissions: Set(); users with modify permissions: Set(livy); groups with modify permissions: Set() 19/08/08 11:50:18 INFO LineBufferedStream: stdout: 19/08/08 11:50:18 INFO Client: Submitting application application_1564675360073_0003 to ResourceManager 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO YarnClientImpl: Submitted application application_1564675360073_0003 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO Client: Application report for application_1564675360073_0003 (state: ACCEPTED) 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO Client: 19/08/08 11:50:19 INFO LineBufferedStream: stdout: client token: N/A 19/08/08 11:50:19 INFO LineBufferedStream: stdout: diagnostics: [Thu Aug 08 11:50:18 +0100 2019] Application is Activated, waiting for resources to be assigned for AM. Last Node which was processed for the application : bda02.vm.altran:45454 ( Partition : [], Total resource : , Available resource : ). Details : AM Partition = ; Partition Resource = ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; Queue's capacity (absolute resource) = ; Queue's used capacity (absolute resource) = ; Queue's max capacity (absolute resource) = ; 19/08/08 11:50:19 INFO LineBufferedStream: stdout: ApplicationMaster host: N/A 19/08/08 11:50:19 INFO LineBufferedStream: stdout: ApplicationMaster RPC port: -1 19/08/08 11:50:19 INFO LineBufferedStream: stdout: queue: default 19/08/08 11:50:19 INFO LineBufferedStream: stdout: start time: 1565261418627 19/08/08 11:50:19 INFO LineBufferedStream: stdout: final status: UNDEFINED 19/08/08 11:50:19 INFO LineBufferedStream: stdout: tracking URL: http://bda01.vm.altran:8088/proxy/application_1564675360073_0003/ 19/08/08 11:50:19 INFO LineBufferedStream: stdout: user: livy 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO ShutdownHookManager: Shutdown hook called 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-5e1fa384-7450-4e34-b150-760102fab560 19/08/08 11:50:19 INFO LineBufferedStream: stdout: 19/08/08 11:50:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-7c7eee5d-d1da-45ea-be2d-d1ab82acca65 19/08/08 11:50:37 INFO RSCClient: Received result for d823f4a0-afaf-4123-9424-d1a3432e68c1 19/08/08 11:50:37 INFO InteractiveSession: Interactive session 1 created [appid: application_1564675360073_0003, owner: null, proxyUser: None, state: idle, kind: shared, info: {driverLogUrl=http://bda02.vm.altran:8042/node/containerlogs/container_e03_1564675360073_0003_01_000001/livy, sparkUiUrl=http://bda01.vm.altran:8088/proxy/application_1564675360073_0003/}] 19/08/08 11:50:47 INFO RSCClient: Received result for 7e055267-a73d-4a50-916e-1a347479fa98 19/08/08 11:50:48 INFO InteractiveSession: Stopping InteractiveSession 1... 19/08/08 11:50:52 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs. 19/08/08 11:50:52 INFO InteractiveSession: Stopped InteractiveSession 1. 19/08/08 11:51:29 INFO InteractiveSession: Stopping InteractiveSession 0... 19/08/08 11:51:29 INFO InteractiveSession: Stopped InteractiveSession 0. 19/08/08 12:40:09 INFO InteractiveSession$: Creating Interactive session 2: [owner: null, request: [kind: shared, proxyUser: None, conf: spark.driver.memory -> 1g,spark.executor.instances -> 1,spark.driver.cores -> 1,livy.uri -> http://10.0.0.157:8999,spark.executor.memory -> 1g,spark.dynamicAllocation.enabled -> false,spark.executor.cores -> 1, heartbeatTimeoutInSecond: 0]] 19/08/08 12:40:09 INFO RpcServer: Connected to the port 10001 19/08/08 12:40:09 WARN RSCConf: Your hostname, bda01.vm.altran, resolves to a loopback address, but we couldn't find any external IP address! 19/08/08 12:40:09 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. 19/08/08 12:40:10 INFO InteractiveSessionManager: Registering new session 2 19/08/08 12:40:10 INFO LineBufferedStream: stdout: Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. 19/08/08 12:40:11 INFO LineBufferedStream: stdout: 19/08/08 12:40:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/08/08 12:40:12 INFO LineBufferedStream: stdout: 19/08/08 12:40:12 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 19/08/08 12:40:12 INFO LineBufferedStream: stdout: 19/08/08 12:40:12 INFO RMProxy: Connecting to ResourceManager at bda01.vm.altran/10.0.0.157:8050 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Requesting a new application from cluster with 1 NodeManagers 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (24576 MB per container) 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Setting up container launch context for our AM 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Setting up the launch environment for our AM container 19/08/08 12:40:13 INFO LineBufferedStream: stdout: 19/08/08 12:40:13 INFO Client: Preparing resources for our AM container 19/08/08 12:40:15 INFO LineBufferedStream: stdout: 19/08/08 12:40:15 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:40:15 INFO LineBufferedStream: stdout: 19/08/08 12:40:15 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:40:15 INFO LineBufferedStream: stdout: 19/08/08 12:40:15 INFO Client: Distribute hdfs cache file as spark.sql.hive.metastore.jars for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:40:15 INFO LineBufferedStream: stdout: 19/08/08 12:40:15 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:40:16 INFO LineBufferedStream: stdout: 19/08/08 12:40:16 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-api-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/livy-api-0.5.0.3.1.0.0-78.jar 19/08/08 12:40:17 INFO LineBufferedStream: stdout: 19/08/08 12:40:17 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-rsc-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/livy-rsc-0.5.0.3.1.0.0-78.jar 19/08/08 12:40:18 INFO LineBufferedStream: stdout: 19/08/08 12:40:18 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/netty-all-4.1.17.Final.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/netty-all-4.1.17.Final.jar 19/08/08 12:40:19 INFO LineBufferedStream: stdout: 19/08/08 12:40:19 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/commons-codec-1.9.jar 19/08/08 12:40:21 INFO LineBufferedStream: stdout: 19/08/08 12:40:21 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-core_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/livy-core_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:40:22 INFO LineBufferedStream: stdout: 19/08/08 12:40:22 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-repl_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/livy-repl_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:40:23 INFO LineBufferedStream: stdout: 19/08/08 12:40:23 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/datanucleus-rdbms-3.2.9.jar 19/08/08 12:40:24 INFO LineBufferedStream: stdout: 19/08/08 12:40:24 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/datanucleus-api-jdo-3.2.6.jar 19/08/08 12:40:25 INFO LineBufferedStream: stdout: 19/08/08 12:40:25 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/datanucleus-core-3.2.10.jar 19/08/08 12:40:26 INFO LineBufferedStream: stdout: 19/08/08 12:40:26 INFO Client: Uploading resource file:/etc/spark2/3.1.0.0-78/0/hive-site.xml -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/hive-site.xml 19/08/08 12:40:27 INFO LineBufferedStream: stdout: 19/08/08 12:40:27 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/R/lib/sparkr.zip#sparkr -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/sparkr.zip 19/08/08 12:40:29 INFO LineBufferedStream: stdout: 19/08/08 12:40:29 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/pyspark.zip 19/08/08 12:40:30 INFO LineBufferedStream: stdout: 19/08/08 12:40:30 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/py4j-0.10.7-src.zip 19/08/08 12:40:32 INFO LineBufferedStream: stdout: 19/08/08 12:40:32 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/pyspark.zip added multiple times to distributed cache 19/08/08 12:40:32 INFO LineBufferedStream: stdout: 19/08/08 12:40:32 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache 19/08/08 12:40:32 INFO LineBufferedStream: stdout: 19/08/08 12:40:32 INFO Client: Uploading resource file:/tmp/spark-abb518bc-e77d-44a1-950a-56c072df334f/__spark_conf__4983130400535366465.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0006/__spark_conf__.zip 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO SecurityManager: Changing view acls to: livy 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO SecurityManager: Changing modify acls to: livy 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO SecurityManager: Changing view acls groups to: 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO SecurityManager: Changing modify acls groups to: 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy); groups with view permissions: Set(); users with modify permissions: Set(livy); groups with modify permissions: Set() 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO Client: Submitting application application_1564675360073_0006 to ResourceManager 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO YarnClientImpl: Submitted application application_1564675360073_0006 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO Client: Application report for application_1564675360073_0006 (state: ACCEPTED) 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO Client: 19/08/08 12:40:34 INFO LineBufferedStream: stdout: client token: N/A 19/08/08 12:40:34 INFO LineBufferedStream: stdout: diagnostics: [Thu Aug 08 12:40:34 +0100 2019] Application is Activated, waiting for resources to be assigned for AM. Last Node which was processed for the application : bda02.vm.altran:45454 ( Partition : [], Total resource : , Available resource : ). Details : AM Partition = ; Partition Resource = ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; Queue's capacity (absolute resource) = ; Queue's used capacity (absolute resource) = ; Queue's max capacity (absolute resource) = ; 19/08/08 12:40:34 INFO LineBufferedStream: stdout: ApplicationMaster host: N/A 19/08/08 12:40:34 INFO LineBufferedStream: stdout: ApplicationMaster RPC port: -1 19/08/08 12:40:34 INFO LineBufferedStream: stdout: queue: default 19/08/08 12:40:34 INFO LineBufferedStream: stdout: start time: 1565264434100 19/08/08 12:40:34 INFO LineBufferedStream: stdout: final status: UNDEFINED 19/08/08 12:40:34 INFO LineBufferedStream: stdout: tracking URL: http://bda01.vm.altran:8088/proxy/application_1564675360073_0006/ 19/08/08 12:40:34 INFO LineBufferedStream: stdout: user: livy 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO ShutdownHookManager: Shutdown hook called 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-6dfebf27-4ba0-4d32-a136-07d28c429836 19/08/08 12:40:34 INFO LineBufferedStream: stdout: 19/08/08 12:40:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-abb518bc-e77d-44a1-950a-56c072df334f 19/08/08 12:40:51 INFO RSCClient: Received result for 321589ae-3e14-4f81-add2-ec5ff6309b77 19/08/08 12:40:51 INFO InteractiveSession: Interactive session 2 created [appid: application_1564675360073_0006, owner: null, proxyUser: None, state: idle, kind: shared, info: {driverLogUrl=http://bda02.vm.altran:8042/node/containerlogs/container_e03_1564675360073_0006_01_000001/livy, sparkUiUrl=http://bda01.vm.altran:8088/proxy/application_1564675360073_0006/}] 19/08/08 12:40:57 INFO RSCClient: Received result for 0e51b29f-80ff-43c2-a85a-d26392372020 19/08/08 12:40:59 INFO InteractiveSession: Stopping InteractiveSession 2... 19/08/08 12:41:03 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs. 19/08/08 12:41:03 INFO InteractiveSession: Stopped InteractiveSession 2. 19/08/08 12:42:57 INFO InteractiveSession$: Creating Interactive session 3: [owner: null, request: [kind: shared, proxyUser: None, conf: spark.driver.memory -> 1g,spark.executor.instances -> 1,spark.driver.cores -> 1,livy.uri -> http://10.0.0.157:8999,spark.executor.memory -> 1g,spark.dynamicAllocation.enabled -> false,spark.executor.cores -> 1, heartbeatTimeoutInSecond: 0]] 19/08/08 12:42:57 INFO RpcServer: Connected to the port 10001 19/08/08 12:42:57 WARN RSCConf: Your hostname, bda01.vm.altran, resolves to a loopback address, but we couldn't find any external IP address! 19/08/08 12:42:57 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. 19/08/08 12:42:58 INFO InteractiveSessionManager: Registering new session 3 19/08/08 12:42:59 INFO LineBufferedStream: stdout: Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. 19/08/08 12:42:59 INFO LineBufferedStream: stdout: 19/08/08 12:42:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/08/08 12:43:00 INFO LineBufferedStream: stdout: 19/08/08 12:43:00 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 19/08/08 12:43:00 INFO LineBufferedStream: stdout: 19/08/08 12:43:00 INFO RMProxy: Connecting to ResourceManager at bda01.vm.altran/10.0.0.157:8050 19/08/08 12:43:00 INFO LineBufferedStream: stdout: 19/08/08 12:43:00 INFO Client: Requesting a new application from cluster with 1 NodeManagers 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (24576 MB per container) 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Client: Setting up container launch context for our AM 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Client: Setting up the launch environment for our AM container 19/08/08 12:43:01 INFO LineBufferedStream: stdout: 19/08/08 12:43:01 INFO Client: Preparing resources for our AM container 19/08/08 12:43:03 INFO LineBufferedStream: stdout: 19/08/08 12:43:03 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:43:03 INFO LineBufferedStream: stdout: 19/08/08 12:43:03 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:43:03 INFO LineBufferedStream: stdout: 19/08/08 12:43:03 INFO Client: Distribute hdfs cache file as spark.sql.hive.metastore.jars for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:43:03 INFO LineBufferedStream: stdout: 19/08/08 12:43:03 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:43:03 INFO LineBufferedStream: stdout: 19/08/08 12:43:03 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-api-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/livy-api-0.5.0.3.1.0.0-78.jar 19/08/08 12:43:04 INFO LineBufferedStream: stdout: 19/08/08 12:43:04 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-rsc-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/livy-rsc-0.5.0.3.1.0.0-78.jar 19/08/08 12:43:06 INFO LineBufferedStream: stdout: 19/08/08 12:43:06 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/netty-all-4.1.17.Final.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/netty-all-4.1.17.Final.jar 19/08/08 12:43:07 INFO LineBufferedStream: stdout: 19/08/08 12:43:07 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/commons-codec-1.9.jar 19/08/08 12:43:09 INFO LineBufferedStream: stdout: 19/08/08 12:43:09 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-core_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/livy-core_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:43:10 INFO LineBufferedStream: stdout: 19/08/08 12:43:10 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-repl_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/livy-repl_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:43:11 INFO LineBufferedStream: stdout: 19/08/08 12:43:11 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/datanucleus-rdbms-3.2.9.jar 19/08/08 12:43:12 INFO LineBufferedStream: stdout: 19/08/08 12:43:12 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/datanucleus-api-jdo-3.2.6.jar 19/08/08 12:43:14 INFO LineBufferedStream: stdout: 19/08/08 12:43:14 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/datanucleus-core-3.2.10.jar 19/08/08 12:43:16 INFO LineBufferedStream: stdout: 19/08/08 12:43:16 INFO Client: Uploading resource file:/etc/spark2/3.1.0.0-78/0/hive-site.xml -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/hive-site.xml 19/08/08 12:43:17 INFO LineBufferedStream: stdout: 19/08/08 12:43:17 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/R/lib/sparkr.zip#sparkr -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/sparkr.zip 19/08/08 12:43:18 INFO LineBufferedStream: stdout: 19/08/08 12:43:18 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/pyspark.zip 19/08/08 12:43:19 INFO LineBufferedStream: stdout: 19/08/08 12:43:19 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/py4j-0.10.7-src.zip 19/08/08 12:43:20 INFO LineBufferedStream: stdout: 19/08/08 12:43:20 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/pyspark.zip added multiple times to distributed cache 19/08/08 12:43:20 INFO LineBufferedStream: stdout: 19/08/08 12:43:20 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache 19/08/08 12:43:21 INFO LineBufferedStream: stdout: 19/08/08 12:43:21 INFO Client: Uploading resource file:/tmp/spark-01a1cb4c-0dc1-4459-9c33-d6a48a865590/__spark_conf__5907731901503632306.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0007/__spark_conf__.zip 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO SecurityManager: Changing view acls to: livy 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO SecurityManager: Changing modify acls to: livy 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO SecurityManager: Changing view acls groups to: 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO SecurityManager: Changing modify acls groups to: 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy); groups with view permissions: Set(); users with modify permissions: Set(livy); groups with modify permissions: Set() 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO Client: Submitting application application_1564675360073_0007 to ResourceManager 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO YarnClientImpl: Submitted application application_1564675360073_0007 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO Client: Application report for application_1564675360073_0007 (state: ACCEPTED) 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO Client: 19/08/08 12:43:22 INFO LineBufferedStream: stdout: client token: N/A 19/08/08 12:43:22 INFO LineBufferedStream: stdout: diagnostics: [Thu Aug 08 12:43:22 +0100 2019] Scheduler has assigned a container for AM, waiting for AM container to be launched 19/08/08 12:43:22 INFO LineBufferedStream: stdout: ApplicationMaster host: N/A 19/08/08 12:43:22 INFO LineBufferedStream: stdout: ApplicationMaster RPC port: -1 19/08/08 12:43:22 INFO LineBufferedStream: stdout: queue: default 19/08/08 12:43:22 INFO LineBufferedStream: stdout: start time: 1565264602542 19/08/08 12:43:22 INFO LineBufferedStream: stdout: final status: UNDEFINED 19/08/08 12:43:22 INFO LineBufferedStream: stdout: tracking URL: http://bda01.vm.altran:8088/proxy/application_1564675360073_0007/ 19/08/08 12:43:22 INFO LineBufferedStream: stdout: user: livy 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO ShutdownHookManager: Shutdown hook called 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-5c445627-0b32-4ee7-b346-5245d7c5feaa 19/08/08 12:43:22 INFO LineBufferedStream: stdout: 19/08/08 12:43:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-01a1cb4c-0dc1-4459-9c33-d6a48a865590 19/08/08 12:43:38 INFO RSCClient: Received result for a49b6f54-61ac-466e-9eb4-4d324f1a6616 19/08/08 12:43:38 INFO InteractiveSession: Interactive session 3 created [appid: application_1564675360073_0007, owner: null, proxyUser: None, state: idle, kind: shared, info: {driverLogUrl=http://bda02.vm.altran:8042/node/containerlogs/container_e03_1564675360073_0007_01_000001/livy, sparkUiUrl=http://bda01.vm.altran:8088/proxy/application_1564675360073_0007/}] 19/08/08 12:43:45 INFO RSCClient: Received result for 12dd7093-c239-4106-a3b6-0109f844662e 19/08/08 12:43:46 INFO InteractiveSession: Stopping InteractiveSession 3... 19/08/08 12:43:50 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs. 19/08/08 12:43:50 INFO InteractiveSession: Stopped InteractiveSession 3. 19/08/08 12:56:41 INFO InteractiveSession$: Creating Interactive session 4: [owner: null, request: [kind: shared, proxyUser: None, conf: spark.driver.memory -> 1g,spark.executor.instances -> 1,spark.driver.cores -> 1,livy.uri -> http://10.0.0.157:8999,spark.executor.memory -> 1g,spark.dynamicAllocation.enabled -> false,spark.executor.cores -> 1, heartbeatTimeoutInSecond: 0]] 19/08/08 12:56:41 INFO RpcServer: Connected to the port 10001 19/08/08 12:56:41 WARN RSCConf: Your hostname, bda01.vm.altran, resolves to a loopback address, but we couldn't find any external IP address! 19/08/08 12:56:41 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. 19/08/08 12:56:43 INFO InteractiveSessionManager: Registering new session 4 19/08/08 12:56:43 INFO LineBufferedStream: stdout: Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. 19/08/08 12:56:43 INFO LineBufferedStream: stdout: 19/08/08 12:56:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO RMProxy: Connecting to ResourceManager at bda01.vm.altran/10.0.0.157:8050 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Requesting a new application from cluster with 1 NodeManagers 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (24576 MB per container) 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Setting up container launch context for our AM 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Setting up the launch environment for our AM container 19/08/08 12:56:44 INFO LineBufferedStream: stdout: 19/08/08 12:56:44 INFO Client: Preparing resources for our AM container 19/08/08 12:56:47 INFO LineBufferedStream: stdout: 19/08/08 12:56:47 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:56:47 INFO LineBufferedStream: stdout: 19/08/08 12:56:47 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-yarn-archive.tar.gz 19/08/08 12:56:47 INFO LineBufferedStream: stdout: 19/08/08 12:56:47 INFO Client: Distribute hdfs cache file as spark.sql.hive.metastore.jars for HDP, hdfsCacheFile:hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:56:47 INFO LineBufferedStream: stdout: 19/08/08 12:56:47 INFO Client: Source and destination file systems are the same. Not copying hdfs://bda01.vm.altran:8020/hdp/apps/3.1.0.0-78/spark2/spark2-hdp-hive-archive.tar.gz 19/08/08 12:56:47 INFO LineBufferedStream: stdout: 19/08/08 12:56:47 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-api-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/livy-api-0.5.0.3.1.0.0-78.jar 19/08/08 12:56:48 INFO LineBufferedStream: stdout: 19/08/08 12:56:48 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/livy-rsc-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/livy-rsc-0.5.0.3.1.0.0-78.jar 19/08/08 12:56:49 INFO LineBufferedStream: stdout: 19/08/08 12:56:49 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/rsc-jars/netty-all-4.1.17.Final.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/netty-all-4.1.17.Final.jar 19/08/08 12:56:50 INFO LineBufferedStream: stdout: 19/08/08 12:56:50 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/commons-codec-1.9.jar 19/08/08 12:56:52 INFO LineBufferedStream: stdout: 19/08/08 12:56:52 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-core_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/livy-core_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:56:53 INFO LineBufferedStream: stdout: 19/08/08 12:56:53 INFO Client: Uploading resource file:/usr/hdp/current/livy2-server/repl_2.11-jars/livy-repl_2.11-0.5.0.3.1.0.0-78.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/livy-repl_2.11-0.5.0.3.1.0.0-78.jar 19/08/08 12:56:55 INFO LineBufferedStream: stdout: 19/08/08 12:56:55 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/datanucleus-rdbms-3.2.9.jar 19/08/08 12:56:56 INFO LineBufferedStream: stdout: 19/08/08 12:56:56 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/datanucleus-api-jdo-3.2.6.jar 19/08/08 12:56:57 INFO LineBufferedStream: stdout: 19/08/08 12:56:57 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/datanucleus-core-3.2.10.jar 19/08/08 12:56:59 INFO LineBufferedStream: stdout: 19/08/08 12:56:59 INFO Client: Uploading resource file:/etc/spark2/3.1.0.0-78/0/hive-site.xml -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/hive-site.xml 19/08/08 12:57:00 INFO LineBufferedStream: stdout: 19/08/08 12:57:00 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/R/lib/sparkr.zip#sparkr -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/sparkr.zip 19/08/08 12:57:00 INFO LineBufferedStream: stdout: 19/08/08 12:57:00 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/pyspark.zip 19/08/08 12:57:01 INFO LineBufferedStream: stdout: 19/08/08 12:57:01 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/py4j-0.10.7-src.zip 19/08/08 12:57:02 INFO LineBufferedStream: stdout: 19/08/08 12:57:02 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/pyspark.zip added multiple times to distributed cache 19/08/08 12:57:02 INFO LineBufferedStream: stdout: 19/08/08 12:57:02 WARN Client: Same name resource file:///usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache 19/08/08 12:57:03 INFO LineBufferedStream: stdout: 19/08/08 12:57:03 INFO Client: Uploading resource file:/tmp/spark-1d811f8d-ca82-426e-904a-48648b4b6b78/__spark_conf__399977775283774728.zip -> hdfs://bda01.vm.altran:8020/user/livy/.sparkStaging/application_1564675360073_0008/__spark_conf__.zip 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO SecurityManager: Changing view acls to: livy 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO SecurityManager: Changing modify acls to: livy 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO SecurityManager: Changing view acls groups to: 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO SecurityManager: Changing modify acls groups to: 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy); groups with view permissions: Set(); users with modify permissions: Set(livy); groups with modify permissions: Set() 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO Client: Submitting application application_1564675360073_0008 to ResourceManager 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO YarnClientImpl: Submitted application application_1564675360073_0008 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO Client: Application report for application_1564675360073_0008 (state: ACCEPTED) 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO Client: 19/08/08 12:57:04 INFO LineBufferedStream: stdout: client token: N/A 19/08/08 12:57:04 INFO LineBufferedStream: stdout: diagnostics: [Thu Aug 08 12:57:04 +0100 2019] Scheduler has assigned a container for AM, waiting for AM container to be launched 19/08/08 12:57:04 INFO LineBufferedStream: stdout: ApplicationMaster host: N/A 19/08/08 12:57:04 INFO LineBufferedStream: stdout: ApplicationMaster RPC port: -1 19/08/08 12:57:04 INFO LineBufferedStream: stdout: queue: default 19/08/08 12:57:04 INFO LineBufferedStream: stdout: start time: 1565265424198 19/08/08 12:57:04 INFO LineBufferedStream: stdout: final status: UNDEFINED 19/08/08 12:57:04 INFO LineBufferedStream: stdout: tracking URL: http://bda01.vm.altran:8088/proxy/application_1564675360073_0008/ 19/08/08 12:57:04 INFO LineBufferedStream: stdout: user: livy 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO ShutdownHookManager: Shutdown hook called 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO ShutdownHookManager: Deleting directory /tmp/spark-1e652828-fbaf-436c-9799-76ba730c0e98 19/08/08 12:57:04 INFO LineBufferedStream: stdout: 19/08/08 12:57:04 INFO ShutdownHookManager: Deleting directory /tmp/spark-1d811f8d-ca82-426e-904a-48648b4b6b78 19/08/08 12:57:21 INFO RSCClient: Received result for 1a310411-b5ce-46bc-91c4-6eeaf091ccbe 19/08/08 12:57:21 INFO InteractiveSession: Interactive session 4 created [appid: application_1564675360073_0008, owner: null, proxyUser: None, state: idle, kind: shared, info: {driverLogUrl=http://bda02.vm.altran:8042/node/containerlogs/container_e03_1564675360073_0008_01_000001/livy, sparkUiUrl=http://bda01.vm.altran:8088/proxy/application_1564675360073_0008/}] 19/08/08 12:57:27 INFO RSCClient: Received result for 7274264b-dddc-490a-a65a-65302d4efbb0 19/08/08 12:57:28 INFO InteractiveSession: Stopping InteractiveSession 4... 19/08/08 12:57:30 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs. 19/08/08 12:57:30 INFO InteractiveSession: Stopped InteractiveSession 4.