NGINX:在access_log中混淆密码 [英] NGINX: Obfuscate password in access_log

问题描述

我要在访问日志中记录$request_body.

I want to log the $request_body in the access logs.

但是某些请求包含一些敏感的JSON字段，例如密码.

But some of the requests have some JSON fields that are sensitive like passwords.

示例:

[2019-03-28] 201 - POST /api/user/add HTTP/1.1 - {\x22email\x22:\[e-mail]\x22,\x22password\x22:\x22myPassword\x22}

是否有一种方法来混淆密码值，以便输出看起来像这样:

Is there a way to obfuscate the password value so the output would look something like this:

[2019-03-28] 201 - POST /api/user/add HTTP/1.1 - {\x22email\x22:\[e-mail]\x22,\x22password\x22:\x22****\x22}

推荐答案

以下是一些正则表达式模式，可用于混淆各种格式的请求正文.

Here are some regex patterns wich can be used for obfuscating request body data in various formats.

当然，您需要做的第一件事是使用 log_format 指令:

Of course the fisrt thing you need to do is to add obfuscated data to log file line format with log_format directive:

log_format custom '$remote_addr - $remote_user [$time_local] '
                    '"$request" "$obfuscated_request_body" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

让我们看一下下面的正文数据格式(假设我们需要混淆的字段是password).

Let's look at the following post body data formats (assuming the field we need to obfuscate is password).

• 请求正文是JSON字符串(通常是REST API请求)

JSON示例:

{"email":"[e-mail]","password":"myPassword"}

转义的JSON字符串:

Escaped JSON string:

{\x22email\x22:\[e-mail]\x22,\x22password\x22:\x22myPassword\x22}

nginx map块:

map $request_body $obfuscated_request_body {
    "~(.*[{,]\\x22password\\x22:\\x22).*?(\\x22[,}].*)" $1********$2;
    default $request_body;
}

• 请求正文是name和value对的JSON数组(由jQuery 返回) serializeArray() 函数)
• Request body is a JSON array of name and value pairs (returned by jQuery serializeArray() function)

JSON示例:

[{"name":"email","value":"[e-mail]"},{"name":"password","value":"myPassword"}]

转义的JSON字符串:

Escaped JSON string:

[{\x22name\x22:\x22email\x22,\x22value\x22:\[e-mail]\x22},{\x22name\x22:\x22password\x22,\x22value\x22:\x22myPassword\x22}]

nginx map块:

map $request_body $obfuscated_request_body {
    "~(.*[\[,]{\\x22name\\x22:\\x22password\\x22,\\x22value\\x22:\\x22).*?(\\x22}[,\]].*)" $1********$2;
    default $request_body;
}

• 请求正文是一个经过urlencoded的字符串(由带有enctype="application/x-www-form-urlencoded"的HTML表单提交)
• Request body is an urlencoded string (submitted by HTML form with enctype="application/x-www-form-urlencoded")

POST正文示例:

login=test%40test.com&password=myPassword

nginx map块: nginx map块:

nginx map block: nginx map block:

map $request_body $obfuscated_request_body {
    ~(^|.*&)(password=)[^&]*(&.*|$) $1$2********$3;
    default $request_body;
}

如果需要混淆多个数据字段，则可以链接多个map转换:

If you need to obfuscate more than one data field, you can chain several map transformations:

log_format custom '$remote_addr - $remote_user [$time_local] '
                  '"$request" "$obfuscated_request_body_2" $status $body_bytes_sent '
                  '"$http_referer" "$http_user_agent"';

map $request_body $obfuscated_request_body_1 {
    "~(.*[{,]\\x22password\\x22:\\x22).*?(\\x22[,}].*)" $1********$2;
    default $request_body;
}

map $obfuscated_request_body_1 $obfuscated_request_body_2 {
    "~(.*[{,]\\x22email\\x22:\\x22).*?(\\x22[,}].*)" $1********$2;
    default $request_body;
}

所有给定的正则表达式只能在log_format nginx指令的escape=default转义模式下使用！如果出于某种原因您需要将此模式更改为escape=json(可从nginx 1.11.8获得)或escape=none(可从nginx 1.13.10获得)，我也为此转义模式构建了正则表达式，但是出于某些奇怪的原因，直到指定 pcre_jit on; 指令(尽管他们通过了其他PCRE测试).对于那些感兴趣的人，这些正则表达式是

All given regexes will be working only with escape=default escaping mode of log_format nginx directive! If for some reason you need to change this mode to escape=json (available from nginx 1.11.8) or escape=none (available from nginx 1.13.10), I built regexes for this escaping modes too, but for some strange reasons couldn't managed them to work with nginx until specifying pcre_jit on; directive (although they pass other PCRE tests). For those who interested, these regexes are

• 用于escape=json转义模式:

map $request_body $obfuscated_request_body {
    "~(.*[{,]\\\"password\\\":\\\")(?:[^\\]|\\{3}\"|\\{2}[bfnrt]|\\{4})*(\\\"[,}].*)" $1********$2;
    default $request_body;
}

用于JSON字符串，

map $request_body $obfuscated_request_body {
    "~(.*[\[,]{\\\"name\\\":\\\"password\\\",\\\"value\\\":\\\")(?:[^\\]|\\{3}\"|\\{2}[bfnrt]|\\{4})*(\\\"}[,\]].*)" $1********$2;
    default $request_body;
}

用于name和value对的JSON数组.

for JSON array of name and value pairs.

• 用于escape=none转义模式:

map $request_body $obfuscated_request_body {
    "~(.*[{,]\"password\":\")(?:[^\\\"]|\\.)*(\"[,}].*)' $1********$2;
    default $request_body;
}

用于JSON字符串，

map $request_body $obfuscated_request_body {
    "~(.*[\[,]{\"name\":\"password\",\"value\":\")(?:[^\\\"]|\\.)*(\"}[,\]].*)" $1********$2;
    default $request_body;
}

用于name和value对的JSON数组.

for JSON array of name and value pairs.

有时，人们还需要混淆作为GET请求查询参数传递的数据.为此，在保留原始nginx访问日志格式的同时，让我们首先看一下默认的访问日志格式:

Sometimes people also need to obfuscate data passed as GET request query parameters. To do this while preserving the original nginx access log format, let's look at the default access log format first:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

nginx bulit-in $request变量可以表示为$request_method $request_uri $server_protocol变量序列:

nginx bulit-in $request variable can be represented as $request_method $request_uri $server_protocol sequence of variables:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request_method $request_uri $server_protocol" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

我们需要混淆$request_uri变量数据的一部分:

We need to obfuscate part of $request_uri variable data:

log_format custom '$remote_addr - $remote_user [$time_local] '
                  '"$request_method $obfuscated_request_uri $server_protocol" $status $body_bytes_sent '
                  '"$http_referer" "$http_user_agent"';

map $request_uri $obfuscated_request_uri {
    ~(.+\?)(.*&)?(password=)[^&]*(&.*|$) $1$2$3********$4;
    default $request_uri;
}

要混淆多个查询参数，您可以如上所述链接多个map翻译.

To obfuscate several query parameters you can chain several map translations as shown above.

Alvin Thompson 评论了OP的问题，其中提到了一些攻击媒介，例如非常大的压缩请求. 值得一提的是，nginx将以压缩形式原样"记录这些请求，因此日志文件不会以不可预测的方式增长.

Alvin Thompson commented OP's question mentioning some attack vectors like very large compressed requests. It is worth mentioning that nginx will log these requests "as-is" in their compressed form, so log files will not grow an unpredictable way.

假设我们的日志文件具有以下格式:

Assuming our log file has following format:

log_format debug '$remote_addr - $remote_user [$time_local] '
                 '"$request" $request_length $content_length '
                 '"$request_body" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';

压缩后的5,000个空格的请求将记录为

request with gzipped body of 5,000 spaces will be logged as

127.0.0.1 - - [09/Feb/2020:05:27:41 +0200] "POST /dump.php HTTP/1.1" 193 41 "\x1F\x8B\x08\x00\x00\x00\x00\x00\x00\x0B\xED\xC11\x01\x00\x00\x00\xC2\xA0*\xEB\x9F\xD2\x14~@\x01\x00\x00\x00\x00o\x03`,\x0B\x87\x88\x13\x00\x00" 200 6881 "-" "curl/7.62.0"

如您所见，$request_length和$content_length值(193和41)反映了来自客户端的传入数据的长度，而不是表示解压缩数据流的字节数.

As you can see, $request_length and $content_length values (193 and 41) reflects the length of the incoming data from the client and not the byte count of the decompressed data stream.

为了过滤异常大的未压缩请求，您还可以按其长度过滤请求正文:

In order to filter abnormally large uncompressed requests, you can additionally filter request bodies by their length:

map $content_length $processed_request_body {
    # Here are some regexes for log filtering by POST body maximum size
    # (only one should be used at a time)

    # Content length value is 4 digits or more ($request_length > 999)
    "~(.*\d{4})" "Too big (request length $1 bytes)";

    # Content length > 499
    "~^((?:[5-9]|\d{2,})\d{2})" "Too big (request length $1 bytes)";

    # Content length > 2999
    "~^((?:[3-9]|\d{2,})\d{3})" "Too big (request length $1 bytes)";

    default $request_body;
}

map $processed_request_body $obfuscated_request_body {
    ...
    default $processed_request_body;
}

这篇关于NGINX:在access_log中混淆密码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！