Spark堆空间和Azure上的简单电子邮件 [英] Spark heap space and simple email on Azure

问题描述

I'm using IntelliJ and Scala to program a Spark job.

我有一个对象.在本地计算机上运行它时，出现有关Java堆大小的错误.我进入IntelliJ设置，就能添加更多空间.

I have an object. When I run it on the local machine I get an error about Java Heap size. I go into IntelliJ settings and I am able to add more space.

从那以后，我在Azure上启动了一个Spark 2.2集群.

I have since spun up a Spark 2.2 cluster on Azure.

当我通过IntelliJ将作业提交给Azure时，在本地运行该作业时出现两个错误

When I submit the job to Azure via IntelliJ, I get two errors that aren't there when running it locally

YARN Diagnostics: User class threw exception: java.lang.OutOfMemoryError: Java heap space

How do I set the Java heap space on Spark sitting on Azure?

YARN Diagnostics: User class threw exception: java.lang.NoClassDefFoundError: 
org/apache/commons/mail/DefaultAuthenticator

I believe the issue is at this line of code:

val email = new SimpleEmail
email.setHostName("smtp.googlemail.com")

email.setSmtpPort(465)
email.setAuthenticator(new DefaultAuthenticator("MY EMAIL Address", "MyPassword"))

How do I send an email from spark on Azure? This code works fine locally. What do I need to do to get this working?

推荐答案

你好，

您可以在Ambari UI中更改YARN堆设置:

You can change YARN heap settings in the Ambari UI:

您的电子邮件问题看起来像是缺少依赖项错误，您可以检查您的类路径或共享提交作业的方式吗?

Your email problem looks like a missing dependency error, could you check your classpath or share how you are submitting the job? 

这篇关于Spark堆空间和Azure上的简单电子邮件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！