如何使spark-submit的输出静音? [英] how to quieten output from spark-submit?

问题描述

我正在尝试针对bluemix上的spark自动执行spark-submit作业，但是我收到了很多输出(例如，文件上传状态).对于手动运行spark-submit来说，此详细输出很好，但是如果尝试从另一个脚本执行，则只是噪音.

I'm trying to automate spark-submit jobs for spark on bluemix, but I'm receiving a lot of output (e.g. file upload status). This verbose output is fine for manually running spark-submit, but it is just noise when trying to execute if from another script.

是否有办法使 bluemix自定义火花提交脚本的输出静默?

Is there a way to quieten the output from the bluemix custom spark-submit script?

更新:

我要使输出安静的输出是从spark-submit脚本生成的，例如:

The output I'm trying to quieten is that generated from the spark-submit script, for example:

Uploading ./truststore.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3103    0    99  100  3004     60   1835  0:00:01  0:00:01 --:--:--  1835

我不是不是试图使spark的输出静默化，因此使 log4j 静默无济于事.

I am not trying to quieten the output from spark, so quietening log4j will not help.

推荐答案

不确定这是否正是您想要的:-

Not sure if this what exactly you want:-

只需将所有内容重定向到/dev/null

Just redirect everything to /dev/null

./spark-submit.sh --vcap凭证/vcap.json-部署模式群集--master https://xxxx:8443 -文件abc.txt hellopy.py file://abc.txt>/dev/null 2>& 1

./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://x.x.x.x:8443 --files abc.txt hellopy.py file://abc.txt > /dev/null 2>&1

谢谢，查尔斯.

这篇关于如何使spark-submit的输出静音?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！