AWS CLI EMR 获取主节点实例 ID 并标记它 [英] AWS CLI EMR get Master node Instance ID and tag it

问题描述

我想自动化集群的运行，并且可以使用标签来获取 EC2 实例的属性，例如它的实例 ID.

I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.

https://docs 上的文档.aws.amazon.com/cli/latest/reference/emr/create-cluster.html 指出

--标签(列表)

与集群关联的标签列表，适用于每个亚马逊集群中的 EC2 实例.标签是键值对，包括最多 128 个字符的必需密钥字符串，以及可选值字符串，最多 256 个字符.

A list of tags to associate with a cluster, which apply to each Amazon EC2 instance in the cluster. Tags are key-value pairs that consist of a required key string with a maximum of 128 characters, and an optional value string with a maximum of 256 characters.

您可以以 key=value 格式指定标签，也可以添加不带标签的标签仅使用键名的值，例如 key .使用空格分隔多个标签.

You can specify tags in key=value format or you can add a tag without a value using only the key name, for example key . Use a space to separate multiple tags.

因此，这会将标签应用于每个 EC2 实例，包括主服务器和从服务器.如何辨别哪个实例是主节点?

So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?

附加信息:我正在使用以下命令根据标签从 aws cli 获取属性，您可以分别用标签键值对替换Name"和Prod".

Additional Info: I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.

aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) |   select(.[].Tags[].Key == "Name") |   {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags}   | [.]' | jq .[].InstanceId

推荐答案

正如您在创建 EMR 集群时所注意到的，所有节点(主、从、任务)的标签都相同.

As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).

您会发现使用 AWS CLI 的这个过程很复杂.我的建议是查看下面的示例，然后编写一个 Python 程序来执行此操作.

You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.

将您自己的标签添加到 EC2 实例的过程.

Process to add your own tags to the EC2 instances.

第 1 步:列出您的 EMR 集群:aws emr 列表集群

STEP 1: List your EMR Clusters: aws emr list-clusters

这将输出 JSON:

{
    "Clusters": [
        {
            "Id": "j-ABCDEFGHIJKLM",
            "Name": "'MyCluster'",
            "Status": {
                "State": "WAITING",
                "StateChangeReason": {
                    "Message": "Cluster ready after last step completed."
                },
                "Timeline": {
                    "CreationDateTime": 1536626095.303,
                    "ReadyDateTime": 1536626568.482
                }
            },
            "NormalizedInstanceHours": 0
        }
    ]
}

第 2 步:记下 JSON 中的集群 ID:

STEP 2: Make a note of the Cluster ID from the JSON:

"Id": "j-ABCDEFGHIJKLM",

第 3 步:描述您的 EMR 集群:aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM

STEP 3: Describe your EMR Cluster: aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM

这将输出 JSON(我已将此输出截断到 MASTER 部分):

This will output JSON (I have truncated this output to just the MASTER section):

{
    "Cluster": {
        "Id": "j-ABCDEFGHIJKLM",
        "Name": "'Test01'",
....
        "InstanceGroups": [
            {
                "Id": "ig-2EHOYXFABCDEF",
                "Name": "Master Instance Group",
                "Market": "ON_DEMAND",
                "InstanceGroupType": "MASTER",
                "InstanceType": "m3.xlarge",
                "RequestedInstanceCount": 1,
                "RunningInstanceCount": 1,
                "Status": {
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    },
                    "Timeline": {
                        "CreationDateTime": 1536626095.316,
                        "ReadyDateTime": 1536626533.886
                    }
                },
                "Configurations": [],
                "EbsBlockDevices": [],
                "ShrinkPolicy": {}
            },
....
        ]
    }
}

第 4 步:InstanceGroups 是一个数组.找到 InstanceGroupType 为 MASTER 的条目.记下 Id.

STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.

"Id": "ig-2EHOYXFABCDEF",

第 5 步:列出您的集群实例:aws emr list-instances --cluster-id j-ABCDEFGHIJKLM

STEP 5: List your cluster instances: aws emr list-instances --cluster-id j-ABCDEFGHIJKLM

这将输出 JSON(我已经截断了输出):

This will output JSON (I have truncated the output):

{
    "Instances": [
....
        {
            "Id": "ci-31LGK4KIECHNY",
            "Ec2InstanceId": "i-0524ec45912345678",
            "PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
            "PublicIpAddress": "52.123.201.221",
            "PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
            "PrivateIpAddress": "172.31.41.111",
            "Status": {
                "State": "RUNNING",
                "StateChangeReason": {},
                "Timeline": {
                    "CreationDateTime": 1536626164.073,
                    "ReadyDateTime": 1536626533.886
                }
            },
            "InstanceGroupId": "ig-2EHOYXFABCDEF",
            "Market": "ON_DEMAND",
            "InstanceType": "m3.xlarge",
            "EbsVolumes": []
        }
    ]
}

第 6 步:找到匹配的 InstanceGroupId ig-2EHOYXFABCDEF.这将为您提供 MASTER 的 EC2 实例 ID:"Ec2InstanceId": "i-0524ec45912345678"

STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"

第 7 步:标记您的 EC2 实例:

Step 7: Tag your EC2 instance:

aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER

使用 CLI 过滤器 和/或 jq 上面的步骤可能更简单，但这应该是足够的信息，以便您知道如何查找和标记 EMR 主实例.

The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.

这篇关于AWS CLI EMR 获取主节点实例 ID 并标记它的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！