如何在 EMR 集群 boostrap 上设置 livy.server.session.timeout? [英] how to set livy.server.session.timeout on EMR cluster boostrap?

问题描述

我正在创建一个 EMR 集群，并使用 jupyter notebook 运行一些 spark 任务.我的任务在执行大约 1 小时后死亡，错误是:

I am creating an EMR cluster, and using jupyter notebook to run some spark tasks. My tasks die after approximately 1 hour of execution, and the error is:

An error was encountered:
Invalid status code '400' from https://xxx.xx.x.xxx:18888/sessions/0/statements/20 with error payload: "requirement failed: Session isn't active."

我的理解是它与 Livy 配置 livy.server.session.timeout 相关，但我不知道如何在集群的引导程序中设置它(我需要在引导程序中执行此操作，因为创建的集群没有 ssh 访问权限)

My understanding is that it is related to the Livy config livy.server.session.timeout, but I don't know how I can set it in the bootstrap of the cluster (I need to do it in the bootstrap because the cluster is created with no ssh access)

先谢谢了

推荐答案

在 EMR 上，livy-conf 是 livy 的 livy.conf 文件的属性分类，因此，在创建 EMR 集群时，选择 advanced options 并选择 Livy 作为要安装的应用程序，请在 Enter Configuration 字段中传递此 EMR 配置.

On EMR, livy-conf is the classification for the properties for livy's livy.conf file, so when creating an EMR cluster, choose advanced options with Livy as an application chosen to install, please pass this EMR configuration in the Enter Configuration field.

[{'classification': 'livy-conf','Properties': {'livy.server.session.timeout':'5h'}}]

• 在 EMR 上，Livy 二进制文件位于 /etc/livy/，因此配置文件位于 /etc/livy/conf/livy.conf莉>
  • On EMR, Livy binary is located at /etc/livy/, and so the config file is at /etc/livy/conf/livy.conf

  为了验证这一点，

  • 使用已知的 ec2 密钥对、Livy 及以上配置创建 EMR 集群
  • 使用 ec2 key-pair，登录到与集群关联的 EC2 主节点 ssh -i some-ec2-key-pair.pem [e-mail]
  • 导航到 /etc/livy/conf，vim livy.conf &查看livy.server.session.timeout
  的更新值
  • Create an EMR cluster with a known ec2 key-pair, Livy and above config
  • Using the ec2 key-pair, login to the EC2 Master node associated with the cluster ssh -i some-ec2-key-pair.pem [e-mail]
  • Navigate to /etc/livy/conf, vim livy.conf & see the updated value of livy.server.session.timeout

  这篇关于如何在 EMR 集群 boostrap 上设置 livy.server.session.timeout?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！