AWS CLI EMR获取主节点实例ID并对其进行标记 [英] AWS CLI EMR get Master node Instance ID and tag it
问题描述
我想自动化集群的运行，并且可以使用标签来获取EC2实例的属性，例如其instance-id.
I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.
https://docs上的文档. aws.amazon.com/cli/latest/reference/emr/create-cluster.html 指出
-标签(列表)
--tags (list)
与集群关联的标签列表，适用于每个亚马逊 集群中的EC2实例.标签是由以下项组成的键值对: 必需的字符串(最多128个字符)和一个 可选的值字符串，最多256个字符.
A list of tags to associate with a cluster, which apply to each Amazon EC2 instance in the cluster. Tags are key-value pairs that consist of a required key string with a maximum of 128 characters, and an optional value string with a maximum of 256 characters.
您可以指定键=值格式的标签，也可以添加不带标签的标签 仅使用键名的值，例如key.使用空格 分隔多个标签.
You can specify tags in key=value format or you can add a tag without a value using only the key name, for example key . Use a space to separate multiple tags.
因此，这会将标签应用于每个EC2实例，包括主服务器和从服务器.如何识别主节点是哪个实例?
So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?
其他信息: 我正在使用以下命令从基于标签的aws cli获取属性，您可以在其中分别用标签键值对替换名称"和产品".
Additional Info: I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.
aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) | select(.[].Tags[].Key == "Name") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq .[].InstanceId
推荐答案
正如您在创建EMR群集时所指出的那样，所有节点(主节点，从节点，任务节点)的标签都相同.
As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).
您会发现使用AWS CLI的此过程很复杂.我的建议是查看下面的示例，然后编写一个Python程序来做到这一点.
You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.
将自己的标签添加到EC2实例的过程.
Process to add your own tags to the EC2 instances.
第1步:列出您的EMR群集:
aws emr list-clusters
STEP 1: List your EMR Clusters:
aws emr list-clusters
这将输出JSON:
{
"Clusters": [
{
"Id": "j-ABCDEFGHIJKLM",
"Name": "'MyCluster'",
"Status": {
"State": "WAITING",
"StateChangeReason": {
"Message": "Cluster ready after last step completed."
},
"Timeline": {
"CreationDateTime": 1536626095.303,
"ReadyDateTime": 1536626568.482
}
},
"NormalizedInstanceHours": 0
}
]
}
第2步:记下JSON中的集群ID:
STEP 2: Make a note of the Cluster ID from the JSON:
"Id": "j-ABCDEFGHIJKLM",
第3步:描述您的EMR集群:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
STEP 3: Describe your EMR Cluster:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
这将输出JSON(我已将该输出截断为MASTER部分):
This will output JSON (I have truncated this output to just the MASTER section):
{
"Cluster": {
"Id": "j-ABCDEFGHIJKLM",
"Name": "'Test01'",
....
"InstanceGroups": [
{
"Id": "ig-2EHOYXFABCDEF",
"Name": "Master Instance Group",
"Market": "ON_DEMAND",
"InstanceGroupType": "MASTER",
"InstanceType": "m3.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
},
"Timeline": {
"CreationDateTime": 1536626095.316,
"ReadyDateTime": 1536626533.886
}
},
"Configurations": [],
"EbsBlockDevices": [],
"ShrinkPolicy": {}
},
....
]
}
}
步骤4:InstanceGroups是一个数组.查找InstanceGroupType是MASTER的条目.记下Id.
STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.
"Id": "ig-2EHOYXFABCDEF",
第5步:列出您的集群实例:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
STEP 5: List your cluster instances:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
这将输出JSON(我已将输出截断了):
This will output JSON (I have truncated the output):
{
"Instances": [
....
{
"Id": "ci-31LGK4KIECHNY",
"Ec2InstanceId": "i-0524ec45912345678",
"PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
"PublicIpAddress": "52.123.201.221",
"PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
"PrivateIpAddress": "172.31.41.111",
"Status": {
"State": "RUNNING",
"StateChangeReason": {},
"Timeline": {
"CreationDateTime": 1536626164.073,
"ReadyDateTime": 1536626533.886
}
},
"InstanceGroupId": "ig-2EHOYXFABCDEF",
"Market": "ON_DEMAND",
"InstanceType": "m3.xlarge",
"EbsVolumes": []
}
]
}
步骤6:查找匹配的InstanceGroupId ig-2EHOYXFABCDEF.这将为您提供MASTER的EC2实例ID:"Ec2InstanceId": "i-0524ec45912345678"
STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"
第7步:标记您的EC2实例:
Step 7: Tag your EC2 instance:
aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER
使用CLI Filters和/或jq进行上述步骤可能会更简单，但这应该是足够的信息，以便您知道如何查找和标记EMR主实例.
The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.
这篇关于AWS CLI EMR获取主节点实例ID并对其进行标记的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
