阿帕奇星火标准错误和标准输出 [英] Apache Spark Stderr and Stdout

问题描述

我通过连接到具有一个主站和两个奴隶火花独立的群集中运行火花1.0.0。我跑wordcount.py火花提交，实际上它从HDFS中读取数据，并把结果写入到HDFS。到目前为止，一切都很好，其结果将正确地写入HDFS。但事情让我担心的是，当我检查标准输出每个工人，它是空的，我不知道是否被假设是空的？我在标准错误得到了以下内容：

有关标准错误日志页面中的一些（APP-20140704174955-0002）

 星火
执行人命令：Java的-cp::
/usr/local/spark-1.0.0/conf：
/usr/local/spark-1.0.0
/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop1.2.1.jar:/usr/local/hadoop/conf
-XX：MaxPermSize参数=128米-Xms512m-Xmx512Morg.apache.spark.executor.CoarseGrainedExecutorBackend
akka.tcp：// @火花大师：54477 /用户/ CoarseGrainedScheduler0SLAVE21
akka.tcp：// @ sparkWorker SLAVE2：41483 /用户/工作者，APP-20140704174955-0002
========================================
14/07/04十七时50分14秒错误CoarseGrainedExecutorBackend：
驱动程序解除关联[akka.tcp：// @ sparkExecutor SLAVE2：33758]  - ＆GT;
[akka.tcp：// @火花大师：54477]解离！关闭。
 

解决方案

星火总是写的一切，甚至信息到stderr。人们似乎这样做是为了阻止标准输出缓冲的消息，造成少predictable记录。这是一个可以接受的做法，当它知道一个应用程序是永远不会在bash脚本中使用，因此特别适用于记录普遍。

I am running spark-1.0.0 by connecting to a spark standalone cluster which has one master and two slaves. I ran wordcount.py by Spark-submit, actually it reads data from HDFS and also write the results into HDFS. So far everything is fine and the results will correctly be written into HDFS. But the thing makes me concern is that when I check Stdout for each worker, it is empty I dont know whether it is suppose to be empty? and I got following in stderr:

stderr log page for Some(app-20140704174955-0002)

Spark 
Executor Command: "java" "-cp" "::
/usr/local/spark-1.0.0/conf:
/usr/local/spark-1.0.0
/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop1.2.1.jar:/usr/local/hadoop/conf" "
-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend
" "akka.tcp://spark@master:54477/user/CoarseGrainedScheduler" "0" "slave2" "1
" "akka.tcp://sparkWorker@slave2:41483/user/Worker" "app-20140704174955-0002"
========================================


14/07/04 17:50:14 ERROR CoarseGrainedExecutorBackend: 
Driver Disassociated [akka.tcp://sparkExecutor@slave2:33758] -> 
[akka.tcp://spark@master:54477] disassociated! Shutting down.

解决方案

Spark always writes everything, even INFO to stderr. People seem to do this to stop stdout buffering messages and causing less predictable logging. It's an acceptable practice when it's known that an application is never going to be used in bash scripting, so especially common for logging.

这篇关于阿帕奇星火标准错误和标准输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！