如何使用pyspark在mongodb中保存数据帧? [英] how to save dataframe in mongodb using pyspark?

问题描述

my_spark = SparkSession \ .builder \ .appName("DGLE") \ 
 config("spark.mongodb.input.uri", "mongodb://127.0.0.1/local.DGLE") \ getOrCreate() 
JDBC.write.format("com.mongodb.spark.sql.DefaultSource").option("spark.mongodb.output.uri", "mongodb://127.0.0.1:27017/local.DGLE").save() 

这是我的代码，不知道哪里出错了，请帮忙

This is my code, do not know where and what is the error, please help

my_spark = SparkSession \ builder \ .appName("DGLE") \ .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/local.DGLE") \ .getOrCreate() 
JDBC.write.format("com.mongodb.spark.sql.DefaultSource").option("spark.mongodb.output.uri", "mongodb://127.0.0.1:27017/local.DGLE").save()

<小时>

推荐答案

从 pyspark 写入 mongodb 的有效方法是使用 MongoDB Spark 连接器.Connector 会将数据转换成 BSON 格式并保存到 mongodb.假设您有一个名为 df 的 spark 数据框，您想将其保存在 mongodb 中.你可以试试:

Efficient way to write into mongodb from pyspark is to use MongoDB Spark Connector. Connector will convert the data into BSON format and save it to mongodb. Let's say you have spark dataframe named df which you want to save in mongodb. You can try:

from pyspark.sql import SparkSession, SQLContext
from pyspark import SparkConf, SparkContext
sc = SparkContext()
spark = SparkSession(sc)   


df.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").option("spark.mongodb.output.uri","mongodb://username:password@server_details:27017/db_name.collection_name?authSource=admin").save()

如果您使用笔记本，请将其写在顶部-

If you are using notebook write this at the top-

%%configure
{"conf": {"spark.jars.packages": "org.mongodb.spark:mongo-spark-connector_2.11:2.3.2"}}

如果您使用 spark-submit 命令:

If you are using spark-submit command:

spark-submit --conf spark.pyspark.python=/usr/bin/anaconda/envs/py35/bin/python3.5 --packages org.mongodb.spark:mongo-spark-connector_2.11:2.3.1 file_name.py

这篇关于如何使用pyspark在mongodb中保存数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！