在静默模式下通过spark-shell执行scala脚本 [英] Execute the scala script through spark-shell in silent mode

问题描述

需要以静默方式通过spark-shell执行scala脚本.当我使用spark-shell -i "file.scala"时，执行后，我进入了scala交互模式.我不想进去.

Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.

我试图执行spark-shell -i"file.scala".但是我不知道如何在静默模式下执行脚本.

I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.

spark-shell -i "file.scala"

执行后，我进入

scala>

我不想进入scala>模式

此问题也与运行终止脚本有关，即由spark-shell -i script.scala > output.txt运行的"scala脚本"会自行停止(内部指令System.exit(0)终止脚本).
请参见有一个很好的例子的这个问题.

This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script).
See this question with a good example.

它还需要一个静默模式" ，它不会污染output.txt .

It also needs a "silent mode", it is expected to not pollute the output.txt.

假设 Spark v2.2 + .

PS:在很多情况下(通常是小的工具和模块/算法测试)， Spark解释器可能比 compiler 更好.编译！"这里不是答案.

PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.

推荐答案

spark-shell -i file.scala使解释器保持打开状态 最后，因此System.exit(0)必须位于脚本的末尾.最合适的解决方案是将代码放在try {}中，并将System.exit(0)放在finally {}部分中.

spark-shell -i file.scala keeps the interpreter open in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.

如果需要日志记录，则可以使用以下内容:

If logging is requiered you can use something like this:

spark-shell < file.scala > test.log 2>&1 &

如果您在编辑文件方面有限制，并且无法添加System.exit(0)，请使用:

If you have limitations on editing file and you can't add System.exit(0), use:

echo :quit | scala-shell -i file.scala

UPD

UPD

如果要抑制输出中除printlns之外的所有内容，则必须关闭spark-shell的日志记录.配置示例位于此处.禁用$SPARK-HOME/conf/log4j.properties中的任何类型的日志记录应只允许您查看pritnlns.但是我不会在printlns中采用这种方法.应该将常规日志记录与log4j一起使用，而不要使用printlns.您可以对其进行配置，以获得与printlns相同的结果.归结为配置模式.此 answer 提供了以下示例一种可以解决您问题的模式.

If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.

这篇关于在静默模式下通过spark-shell执行scala脚本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！