使用多个 S3 帐户运行 EMR Spark [英] Running EMR Spark With Multiple S3 Accounts

问题描述

我有一个 EMR Spark 作业，需要从一个帐户的 S3 读取数据并写入另一个帐户.
我把我的工作分成两步.

I have an EMR Spark Job that needs to read data from S3 on one account and write to another.
I split my job into two steps.

1. 从 S3 读取数据(不需要凭据，因为我的 EMR 集群在同一个帐户中).

1. read data from the S3 (no credentials required because my EMR cluster is in the same account).

读取第 1 步创建的本地 HDFS 中的数据并将其写入另一个帐户中的 S3 存储桶.

read data in the local HDFS created by step 1 and write it to an S3 bucket in another account.

我已尝试设置 hadoopConfiguration:

sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "<your access key>")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey","<your secretkey>")

并导出集群上的密钥:

$ export AWS_SECRET_ACCESS_KEY=
$ export AWS_ACCESS_KEY_ID=

我尝试了集群和客户端模式以及spark-shell，但都没有成功.

I've tried both cluster and client mode as well as spark-shell with no luck.

每个都返回一个错误:

ERROR ApplicationMaster: User class threw exception: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: 
Access Denied

推荐答案

解决方案其实很简单.

首先，EMR 集群有两个作用:

Firstly, EMR clusters have two roles:

• 授予 EMR 服务权限的服务角色 (EMR_DefaultRole)(例如，用于启动 Amazon EC2 实例)
• 一个 EC2 角色 (EMR_EC2_DefaultRole) 附加到集群中启动的 EC2 实例，使它们能够访问 AWS 凭证(请参阅 使用 IAM 角色向在 Amazon EC2 实例上运行的应用程序授予权限)莉>

• A service role (EMR_DefaultRole) that grants permissions to the EMR service (eg for launching Amazon EC2 instances)
• An EC2 role (EMR_EC2_DefaultRole) that is attached to EC2 instances launched in the cluster, giving them access to AWS credentials (see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances)

这些角色的解释在:默认 IAMAmazon EMR 的角色

因此，集群中启动的每个 EC2 实例都被分配了 EMR_EC2_DefaultRole 角色，该角色通过实例元数据服务提供临时凭证.(有关其工作原理的说明，请参阅:Amazon EC2 的 IAM 角色.)Amazon EMR 节点使用这些凭证访问 AWS 服务，例如 S3、SNS、SQS、CloudWatch 和 DynamoDB.

Therefore, each EC2 instance launched in the cluster is assigned the EMR_EC2_DefaultRole role, which makes temporary credentials available via the Instance Metadata service. (For an explanation of how this works, see: IAM Roles for Amazon EC2.) Amazon EMR nodes use these credentials to access AWS services such as S3, SNS, SQS, CloudWatch and DynamoDB.

其次，您需要向另一个账户中的 Amazon S3 存储桶添加权限，以允许通过 EMR_EC2_DefaultRole 角色进行访问.这可以通过将 bucket policy 添加到 S3 存储桶(此处命名为 other-account-bucket)来完成，如下所示:

Secondly, you will need to add permissions to the Amazon S3 bucket in the other account to permit access via the EMR_EC2_DefaultRole role. This can be done by adding a bucket policy to the S3 bucket (here named other-account-bucket) like this:

{
    "Id": "Policy1",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1",
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::other-account-bucket",
                "arn:aws:s3:::other-account-bucket/*"
            ],
            "Principal": {
                "AWS": [
                    "arn:aws:iam::ACCOUNT-NUMBER:role/EMR_EC2_DefaultRole"
                ]
            }
        }
    ]
}

此策略将所有 S3 权限 (s3:*) 授予属于与 ACCOUNT-NUMBER 匹配的帐户的 EMR_EC2_DefaultRole 角色策略，应该是启动 EMR 集群的账户.授予此类权限时要小心——您可能只想授予 GetObject 权限，而不是授予所有 S3 权限.

This policy grants all S3 permissions (s3:*) to the EMR_EC2_DefaultRole role that belongs to the account matching the ACCOUNT-NUMBER in the policy, which should be the account in which the EMR cluster was launched. Be careful when granting such permissions -- you might want to grant permissions only to GetObject rather than granting all S3 permissions.

就是这样！另一个账户中的存储桶现在将接受来自 EMR 节点的请求，因为它们使用 EMR_EC2_DefaultRole 角色.

That's all! The bucket in the other account will now accept requests from the EMR nodes because they are using the EMR_EC2_DefaultRole role.

免责声明:我通过在账户 A 中创建一个存储桶并为账户 B 中的角色分配权限(如上所示)来测试上述内容.使用该角色在 Account-B 中启动了一个 EC2 实例.我能够通过 AWS 命令​​行界面 (CLI) 从 EC2 实例访问存储桶.我没有在 EMR 中测试它，但是它应该以相同的方式工作.

Disclaimer: I tested the above by creating a bucket in Account-A and assigning permissions (as shown above) to a role in Account-B. An EC2 instance was launched in Account-B with that role. I was able to access the bucket from the EC2 instance via the AWS Command-Line Interface (CLI). I did not test it within EMR, however it should work the same way.

这篇关于使用多个 S3 帐户运行 EMR Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！