用于连接 AWS Cloudwatch Logs、Kinesis Firehose、S3 和 ElasticSearch 的 AWS IAM 策略 [英] AWS IAM Policies to connect AWS Cloudwatch Logs, Kinesis Firehose, S3 and ElasticSearch
问题描述
我正在尝试通过 Kinesis Firehose 将 AWS cloudwatch 日志流式传输到 ES.下面的 terraform 代码给出了一个错误.有什么建议..错误是:
I am trying to stream the AWS cloudwatch logs to ES via Kinesis Firehose. Below terraform code is giving an error. Any suggestions.. The error is:
- aws_cloudwatch_log_subscription_filter.test_kinesis_logfilter:发生 1 个错误:
- aws_cloudwatch_log_subscription_filter.test_kinesis_logfilter:InvalidParameterException:无法将测试消息传递到指定的 Firehose 流.检查给定的 Firehose 流是否处于 ACTIVE 状态.
<代码>
resource "aws_s3_bucket" "bucket" {
bucket = "cw-kinesis-es-bucket"
acl = "private"
}
resource "aws_iam_role" "firehose_role" {
name = "firehose_test_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_elasticsearch_domain" "es" {
domain_name = "firehose-es-test"
elasticsearch_version = "1.5"
cluster_config {
instance_type = "t2.micro.elasticsearch"
}
ebs_options {
ebs_enabled = true
volume_size = 10
}
advanced_options {
"rest.action.multi.allow_explicit_index" = "true"
}
access_policies = <<CONFIG
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal": "*",
"Effect": "Allow",
"Condition": {
"IpAddress": {"aws:SourceIp": ["xxxxx"]}
}
}
]
}
CONFIG
snapshot_options {
automated_snapshot_start_hour = 23
}
tags {
Domain = "TestDomain"
}
}
resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
name = "terraform-kinesis-firehose-test-stream"
destination = "elasticsearch"
s3_configuration {
role_arn = "${aws_iam_role.firehose_role.arn}"
bucket_arn = "${aws_s3_bucket.bucket.arn}"
buffer_size = 10
buffer_interval = 400
compression_format = "GZIP"
}
elasticsearch_configuration {
domain_arn = "${aws_elasticsearch_domain.es.arn}"
role_arn = "${aws_iam_role.firehose_role.arn}"
index_name = "test"
type_name = "test"
}
}
resource "aws_iam_role" "iam_for_lambda" {
name = "iam_for_lambda"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_cloudwatch_log_subscription_filter" "test_kinesis_logfilter" {
name = "test_kinesis_logfilter"
role_arn = "${aws_iam_role.iam_for_lambda.arn}"
log_group_name = "loggorup.log"
filter_pattern = ""
destination_arn = "${aws_kinesis_firehose_delivery_stream.test_stream.arn}"
}
推荐答案
在此配置中，您将指示 Cloudwatch Logs 将日志记录发送到 Kinesis Firehose，而 Kinesis Firehose 又被配置为将接收到的数据写入 S3 和 ElasticSearch.因此，您使用的 AWS 服务之间的通信方式如下:
In this configuration you are directing Cloudwatch Logs to send log records to Kinesis Firehose, which is in turn configured to write the data it receives to both S3 and ElasticSearch. Thus the AWS services you are using are talking to each other as follows:
为了让一个 AWS 服务与另一个服务通信，第一个服务必须承担一个授予其访问权限的角色.在 IAM 术语中，担任角色"意味着临时使用授予该角色的权限行事.AWS IAM 角色有两个关键部分:
In order for one AWS service to talk to another the first service must assume a role that grants it access to do so. In IAM terminology, "assuming a role" means to temporarily act with the privileges granted to that role. An AWS IAM role has two key parts:
- 担任角色政策，控制哪些服务和/或用户可以担任该角色.
- 控制角色授予访问权限的政策.这决定了服务或用户在担任该角色后可以执行的操作.
- The assume role policy, that controls which services and/or users may assume the role.
- The policies controlling what the role grants access to. This decides what a service or user can do once it has assumed the role.
这里需要两个不同的角色.一个角色将授予 Cloudwatch Logs 与 Kinesis Firehose 对话的权限，而第二个角色将授予 Kinesis Firehose 与 S3 和 ElasticSearch 对话的权限.
Two separate roles are needed here. One role will grant Cloudwatch Logs access to talk to Kinesis Firehose, while the second will grant Kinesis Firehose access to talk to both S3 and ElasticSearch.
对于本答案的其余部分，我将假设 Terraform 以对 AWS 账户具有完全管理访问权限的用户身份运行.如果不是这样，首先需要确保 Terraform 作为有权创建和传递角色的 IAM 委托人运行.
For the rest of this answer, I will assume that Terraform is running as a user with full administrative access to an AWS account. If this is not true, it will first be necessary to ensure that Terraform is running as an IAM principal that has access to create and pass roles.
在问题中给出的示例中，aws_cloudwatch_log_subscription_filter 有一个 role_arn，其 assume_role_policy 用于 AWS Lambda，因此 Cloudwatch Logs 没有有权担任此角色.
In the example given in the question, the aws_cloudwatch_log_subscription_filter has a role_arn whose assume_role_policy is for AWS Lambda, so Cloudwatch Logs does not have access to assume this role.
要解决此问题，可以将承担角色策略更改为使用 Cloudwatch Logs 的服务名称:
To fix this, the assume role policy can be changed to use the service name for Cloudwatch Logs:
resource "aws_iam_role" "cloudwatch_logs" {
name = "cloudwatch_logs_to_firehose"
assume_role_policy = jsonencode({
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "logs.us-east-1.amazonaws.com"
},
"Effect": "Allow",
"Sid": "",
},
],
})
}
以上内容允许 Cloudwatch Logs 服务担任该角色.现在该角色需要一个允许写入 Firehose 传输流的访问策略:
The above permits the Cloudwatch Logs service to assume the role. Now the role needs an access policy that permits writing to the Firehose Delivery Stream:
resource "aws_iam_role_policy" "cloudwatch_logs" {
role = aws_iam_role.cloudwatch_logs.name
policy = jsonencode({
"Statement": [
{
"Effect": "Allow",
"Action": ["firehose:*"],
"Resource": [aws_kinesis_firehose_delivery_stream.test_stream.arn],
},
],
})
}
只要 Cloudwatch Logs 服务针对由此 Terraform 配置创建的特定交付流，上述内容就可以调用 任何 Kinesis Firehose 操作.这比严格必要的访问更多；有关更多信息，请参阅 Amazon Kinesis Firehose 的操作和条件上下文键.
The above grants the Cloudwatch Logs service access to call into any Kinesis Firehose action as long as it targets the specific delivery stream created by this Terraform configuration. This is more access than is strictly necessary; for more information, see Actions and Condition Context Keys for Amazon Kinesis Firehose.
要完成此操作，必须更新 aws_cloudwatch_log_subscription_filter 资源以引用此新角色:
To complete this, the aws_cloudwatch_log_subscription_filter resource must be updated to refer to this new role:
resource "aws_cloudwatch_log_subscription_filter" "test_kinesis_logfilter" {
name = "test_kinesis_logfilter"
role_arn = aws_iam_role.cloudwatch_logs.arn
log_group_name = "loggorup.log"
filter_pattern = ""
destination_arn = aws_kinesis_firehose_delivery_stream.test_stream.arn
# Wait until the role has required access before creating
depends_on = aws_iam_role_policy.cloudwatch_logs
}
不幸的是，由于 AWS IAM 的内部设计，在 Terraform 提交策略更改后，通常需要几分钟才能生效，因此在尝试使用政策本身创建后不久.在这种情况下，通常只需等待 10 分钟，然后再次运行 Terraform，此时它应该从中断的地方恢复并重试创建资源.
Unfortunately due to the internal design of AWS IAM, it can often take several minutes for a policy change to come into effect after Terraform submits it, so sometimes a policy-related error will occur when trying to create a new resource using a policy very soon after the policy itself was created. In this case, it's often sufficient to simply wait 10 minutes and then run Terraform again, at which point it should resume where it left off and retry creating the resource.
问题中给出的示例已经有一个 IAM 角色和适用于 Kinesis Firehose 的代入角色策略:
The example given in the question already has an IAM role with a suitable assume role policy for Kinesis Firehose:
resource "aws_iam_role" "firehose_role" {
name = "firehose_test_role"
assume_role_policy = jsonencode({
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
})
}
以上内容授予 Kinesis Firehose 访问权限以担任此角色.和以前一样，该角色还需要一个访问策略来授予该角色的用户访问目标 S3 存储桶的权限:
The above grants Kinesis Firehose access to assume this role. As before, this role also needs an access policy to grant users of the role access to the target S3 bucket:
resource "aws_iam_role_policy" "firehose_role" {
role = aws_iam_role.firehose_role.name
policy = jsonencode({
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": [aws_s3_bucket.bucket.arn]
},
{
"Effect": "Allow",
"Action": ["es:ESHttpGet"],
"Resource": ["${aws_elasticsearch_domain.es.arn}/*"]
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:*:*:log-group:*:log-stream:*"
]
},
],
})
}
上述策略允许 Kinesis Firehose 对创建的 S3 存储桶执行任何操作，对创建的 ElasticSearch 域执行任何操作，并将日志事件写入 Cloudwatch Logs 中的任何日志流.最后一部分并不是绝对必要的，但如果为 Firehose 传输流启用了日志记录，则很重要，否则 Kinesis Firehose 无法将日志写回 Cloudwatch Logs.
The above policy allows Kinesis Firehose to perform any action on the created S3 bucket, any action on the created ElasticSearch domain, and to write log events into any log stream in Cloudwatch Logs. The final part of this is not strictly necessary, but is important if logging is enabled for the Firehose Delivery Stream, or else Kinesis Firehose is unable to write logs back to Cloudwatch Logs.
同样，这比严格必要的访问权限更多.有关支持的特定操作的更多信息，请参阅以下参考资料:
Again, this is more access than strictly necessary. For more information on the specific actions supported, see the following references:
由于这个单一角色有权写入 S3 和 ElasticSearch，因此可以在 Kinesis Firehose 交付流中为这两种交付配置指定它:
Since this single role has access to write to both S3 and to ElasticSearch, it can be specified for both of these delivery configurations in the Kinesis Firehose delivery stream:
resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
name = "terraform-kinesis-firehose-test-stream"
destination = "elasticsearch"
s3_configuration {
role_arn = aws_iam_role.firehose_role.arn
bucket_arn = aws_s3_bucket.bucket.arn
buffer_size = 10
buffer_interval = 400
compression_format = "GZIP"
}
elasticsearch_configuration {
domain_arn = aws_elasticsearch_domain.es.arn
role_arn = aws_iam_role.firehose_role.arn
index_name = "test"
type_name = "test"
}
# Wait until access has been granted before creating the firehose
# delivery stream.
depends_on = [aws_iam_role_policy.firehose_role]
}
<小时>
在完成上述所有布线后，服务应该具有连接此交付管道各部分所需的访问权限.
With all of the above wiring complete, the services should have the access they need to connect the parts of this delivery pipeline.
同样的一般模式适用于两个 AWS 服务之间的任何连接.每个案例所需的重要信息是:
This same general pattern applies to any connection between two AWS services. The important information needed for each case is:
- 将发起请求的服务的服务名称，例如
logs.us-east-1.amazonaws.com或firehose.amazonaws.com.遗憾的是，这些文件通常记录不充分且很难找到，但通常可以在每个服务的用户指南中的政策示例中找到. - 需要授予的操作的名称.可以在 AWS 服务中找到每个服务的完整操作集用于 IAM 策略的操作和条件上下文键.不幸的是，给定的服务到服务集成需要具体哪些操作的文档通常相当缺乏，但在简单的环境中(尽管有任何严格的监管要求或围绕访问的组织政策)通常就足够了使用上述示例中使用的通配符语法授予对给定服务的所有操作的访问权限.
- The service name for the service that will initiate the requests, such as
logs.us-east-1.amazonaws.comorfirehose.amazonaws.com. These are unfortunately generally poorly documented and hard to find, but can usually be found in policy examples within each service's user guide. - The names of the actions that need to be granted. The full set of actions for each service can be found in AWS Service Actions and Condition Context Keys for Use in IAM Policies. Unfortunately again the documentation for specifically which actions are required for a given service-to-service integration is generally rather lacking, but in simple environments (notwithstanding any hard regulatory requirements or organizational policies around access) it usually suffices to grant access to all actions for a given service, using the wildcard syntax used in the above examples.
这篇关于用于连接 AWS Cloudwatch Logs、Kinesis Firehose、S3 和 ElasticSearch 的 AWS IAM 策略的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
