使用AWS CLI将日志流转为弹性 [英] Stream logs to elastic using aws cli

问题描述

我想启用从Cloudwatch到Elasticsearch的Stream to Amazon Elasticsearch Service。

I would like to enable the Stream to Amazon Elasticsearch Service from Cloudwatch to Elasticsearch.

我熟悉如何手动执行此操作，我正在寻找一种通过运行aws cli命令来实现此目标的方法。

I'm familiar with how to do that manually, I'm looking for a way to achieve that by running aws cli commands.

假设Elasticsearch已配置，有什么方法可以自动执行该过程？

assuming Elasticsearch is already configured, is there any way to automate the process ?

推荐答案

更新：

如果您正在使用cloudformation，请在此处查看我的答案。

If you are using cloudformation, take a look at my answer here.

非常感谢@Adiii为我指明了正确的方向，在下面找到了首尾此问题的解决方案。

Many thanks to @Adiii that pointed me in the right direction,Find bellow the end to end solution for this issue.

该解决方案包括以下部分：

The solution Include the following parts :

• create-lambda-role

• create-lambda-role

grant-permissions-lambda-role

grant-permissions-to-lambda-role

I假定lambda函数已经打包并且可以访问。
您可以在此处找到lambda函数>。

I assume that the lambda function is already packaged and accessible. You can find the lambda function here.

更新 var endpoint = $ {Elasticsearch_Endpoint}; 在index.js中，并带有您的Elasticseatch网址，例如-- search-xxx-yyyy.eu-west-1.es.amazonaws.com; 。

Update var endpoint = ${Elasticsearch_Endpoint}; in index.js with your Elasticseatch url e.g - search-xxx-yyyy.eu-west-1.es.amazonaws.com;.

手动操作此处。

1。 create-lambda-role

首先，我们需要创建lambda函数使用的角色，稍后我们将对该角色附加相关政策。

First, we need to create the role that the lambda function use, later we will attach relevant policies to that role.

    cat > lambda-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
  }
} 
EOF

    aws iam create-role \
    --role-name ${ROLE_NAME} \
    --assume-role-policy-document file://lambda-policy.json \
    --profile ${PROFILE} \
    >/dev/null

2。授予Lambda角色的权限

将相关政策附加到Lambda角色。

Attach relevant policies to lambda role.

 cat > lambda-to-es-via-vpc-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1569580358341",
      "Action": "es:*",
      "Effect": "Allow",
      "Resource": "arn:aws:es:${AWS_REGION}:${AWS_ACCOUNT_ID}:domain/${ES_DOMAIN}/*"
    },
    {
      "Sid": "Stmt1569580707924",
      "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF

    aws iam put-role-policy \
    --role-name ${ROLE_NAME} \
    --policy-name lambda-to-es-via-vpc-policy \
    --policy-document file://lambda-to-es-via-vpc-policy.json \
    --profile ${PROFILE} \
    >/dev/null

3。 create-lambda

aws lambda create-function \
    --function-name ${LAMBDA_NAME} \
    --runtime nodejs8.10 \
    --role arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME} \
    --handler ${LAMBDA_NAME}.handler \
    --zip-file fileb://${LAMBDA_NAME}.zip \
    --timeout 30 \
    --vpc-config SubnetIds=${SUBNET_IDS},SecurityGroupIds=${SECURITY_GROUP_IDS} \
    --profile ${PROFILE} \
    >/dev/null

4。 grant-cloudwatch-permission-to-execute-lambda

授予AWS服务或其他帐户使用功能的权限。

Grants an AWS service or another account permission to use a function.

aws lambda add-permission \
--function-name "${LAMBDA_NAME}" \
--statement-id "${LAMBDA_NAME}" \
--principal "logs.${AWS_REGION}.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:/aws/eks/${EKS_CLUSTER}/cluster:*" \
--source-account ${AWS_ACCOUNT_ID} \
--profile ${PROFILE} \
>/dev/null

5。 add-subscription-to-cloudwatch-log-group

创建或更新订阅过滤器并将其与指定的日志组关联。订阅过滤器使您可以订阅日志事件的实时流，并将其传递到特定的目的地。

Creates or updates a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events, and have them delivered to a specific destination.

  aws logs put-subscription-filter \
   --log-group-name  "/aws/eks/${EKS_CLUSTER}/cluster" \
   --filter-name "Common Log Format" \
   --filter-pattern "[host, ident, authuser, date, request, status, bytes]" \
   --destination-arn  arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:${LAMBDA_NAME} \
   --profile ${PROFILE} \
   >/dev/null

这篇关于使用AWS CLI将日志流转为弹性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！