如何将spark-submit的整个输出重定向到文件 [英] How to redirect entire output of spark-submit to a file

问题描述

因此，我试图将apache spark-submit命令的输出重定向到文本文件，但是某些输出无法填充文件.这是我正在使用的命令:

So, I am trying to redirect the output of an apache spark-submit command to text file but some output fails to populate file. Here is the command I am using:

spark-submit something.py > results.txt

我可以在终端中看到输出，但是在文件中看不到它.我在这里忘记或做错了什么?

I can see the output in the terminal but I do not see it in the file. What am I forgetting or doing wrong here?

如果我使用

spark-submit something.py | less

我可以看到所有输出都已输送到less

I can see all the output being piped into less

推荐答案

spark-submit将大部分输出打印到STDERR

spark-submit prints most of it's output to STDERR

要将整个输出重定向到一个文件，可以使用:

To redirect the entire output to one file, you can use:

spark-submit something.py > results.txt 2>&1

或

spark-submit something.py &> results.txt

这篇关于如何将spark-submit的整个输出重定向到文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！