如何使用spark-submit提交查询或包含查询的.sql文件 [英] How to Submit a Query or a .sql file containing a query using spark-submit

问题描述

我想在spark上运行查询.我把它们全部放在我的.sql文件中

I want to run queries on spark. I have them all on my .sql files

目前我的跑步方式是:

spark-sql --master yarn /home/myuser/query.sql

但是一旦查询完成执行，它就会出现在 spark-sql> ...中. spark-sql> 外壳处于活动状态，因此我必须明确地执行 CTRL + C 才能退出外壳.

But as soon as the query completes the execution it comes inside the spark-sql> .... shell prompt And then the job status doesn't changes to Finished Successfully it shows RUNNING becoz the spark-sql> shell is active so i have to explicitly do CTRL+C to come out of the shell.

因此，基本上，我只想提交一个spark作业，它实际上是使用shell脚本文件进行的查询，而没有进入任何spark特定的shell.

SO Basically i just want to submit a spark job which is actually a query using a shell script file and without getting inside any spark specific shell.

我不想在类文件中编写查询，然后构建一个jar，然后提交.

I don't want to write a query in a class file and then build a jar and then submit.

推荐答案

只是发现-f选项对我有用.

Just found out that -f option does the job for me.

spark-sql --master yarn -f /home/myuser/query.sql

这篇关于如何使用spark-submit提交查询或包含查询的.sql文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！