如何使用 spark-submit 提交查询或包含查询的 .sql 文件 [英] How to Submit a Query or a .sql file containing a query using spark-submit

问题描述

我想在 spark 上运行查询.我的 .sql 文件中都有它们

I want to run queries on spark. I have them all on my .sql files

目前我的跑步方式是:

spark-sql --master yarn /home/myuser/query.sql

但是一旦查询完成执行，它就会进入 spark-sql> .... shell 提示然后作业状态不会更改为 Finished Successfully 它显示 RUNNING becozspark-sql> shell 处于活动状态，所以我必须明确地执行 CTRL+C 才能退出 shell.

But as soon as the query completes the execution it comes inside the spark-sql> .... shell prompt And then the job status doesn't changes to Finished Successfully it shows RUNNING becoz the spark-sql> shell is active so i have to explicitly do CTRL+C to come out of the shell.

所以基本上我只想提交一个火花作业，它实际上是一个使用 shell 脚本文件的查询，并且没有进入任何火花特定的 shell.

SO Basically i just want to submit a spark job which is actually a query using a shell script file and without getting inside any spark specific shell.

我不想在类文件中写一个查询然后构建一个jar然后提交.

I don't want to write a query in a class file and then build a jar and then submit.

推荐答案

刚刚发现 -f 选项对我有用.

Just found out that -f option does the job for me.

spark-sql --master yarn -f /home/myuser/query.sql

这篇关于如何使用 spark-submit 提交查询或包含查询的 .sql 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！