阅读Sparkcontext对象的应用程序配置 [英] Read application configuration from Sparkcontext Object

问题描述

我正在开发使用pyspark壳星火应用。

I am developing a Spark application using pyspark shell.

我用下面的命令时，看到的此处我如何创建配置文件：

I kickstarted the iPython notebook service using the command below, see here how I created the profile:

IPYTHON_OPTS="notebook --port 8889 --profile pyspark" pyspark

根据文档上，有一个 SC 已为我创造了一些默认的配置火花上下文对象。

Based on the documentation, there is a sc spark context object already created for me with some default configuration.

在PySpark外壳，特殊的跨preTER感知SparkContext是
  已经为您创建，在​​变量称为SC。制作自己的
  SparkContext将无法正常工作。

"In the PySpark shell, a special interpreter-aware SparkContext is already created for you, in the variable called sc. Making your own SparkContext will not work."

我基本上有两个问题在这里：

I basically have two questions here:

（1）我怎样才能获得配置的摘要默认 SC 对象吗？
我想知道有多少内存已分配，我多少个核心可以使用...等。不过，我只找到了一个名为 getLocalProperty 对象从SC <方法href=\"http://spark.apache.org/docs/1.0.1/api/python/pyspark.context.SparkContext-class.html#setLocalProperty\"相对=nofollow>不知道什么是我应该叫关键参数pyspark API 。

(1) How can I get a summary of the configuration for the default sc object? I want to know how much memory has been allocated, how many cores I can use...etc. However, I only found a method called getLocalProperty for object sc from pyspark API without knowing what is the key argument that I should call.

（2）是否有可能修改sparkcontext与iPythonnotebook工作。如果你不能修改的配置，一旦你开始了IPython的笔记本，如果有一个文件某处某处配置SC？

(2) Is it possible to modify the sparkcontext working with iPythonnotebook. If you cannot modify the configurations once you started the iPython notebook, if there a file somewhere to configure the sc somewhere?

我是相当新的火花，更多的信息（资源），你可以提供，更好的那样。谢谢！

I am fairly new to Spark, the more information(resource) you can provide, the better it would be. Thanks!

推荐答案

它使用pyspark不要求：你可以导入pyspark类，然后实例化SparkContext自己

It is not required to use pyspark: you can import the pyspark classes and then instantiate the SparkContext yourself

from pyspark import SparkContext, SparkConf

设置您的自定义配置：

Set up your custom config:

conf = SparkConf().setAppName(appName).setMaster(master)
# set values into conf here ..
sc = SparkContext(conf=conf)

您可能也想看看普通spark-env.sh

You may also want to look at the general spark-env.sh

CONF /火花env.sh.template＃复制到conf / spark-env.sh，然后修改丘壑作为对你有用

conf/spark-env.sh.template # copy to conf/spark-env.sh and then modify vals as useful to you

如。一些值可能会定义：

eg. some of the values you may customize:

# Options read when launching programs locally with
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append

这篇关于阅读Sparkcontext对象的应用程序配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！