用于访问登录 hive serde 的正则表达式 [英] regex for access log in hive serde

问题描述

我想从访问日志中提取(ip、requestUrl、timeStamp)以加载到 hive 数据库中.访问日志中的一行如下.

I want to extract out (ip, requestUrl, timeStamp) from the access logs to load to hive database. One line from access log is as follows.

66.249.68.6 - - [14/Jan/2012:06:25:03 -0800] "GET /example.com HTTP/1.1" 200 708 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

我尝试了以下正则表达式和几种变体，但没有成功.(加载的表的所有 NULL 值表明正则表达式与输入不匹配).

I tried with following and several variations of regex without any success. (The loaded table is with all NULL values indicating the regex doesn't match the input).

CREATE TABLE access_log (
  remote_ip STRING,
  request_date STRING,
  method STRING,
  request STRING,
  protocol STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES  (
"input.regex" = "([^ ]) . . [([^]]+)] "([^ ]) ([^ ]) ([^ "])" *",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s"
)
STORED AS TEXTFILE;

我对正则表达式不是很有经验.有人可以帮我吗?

I am not very experienced with regex. Can anybody help me with this?

推荐答案

我使用 rubular 来测试我的正则表达式.你也可以用这个表达

I use rubular to test my regex. You can also use this expression

([^ ]*) ([^ ]*) ([^ ]*) (?:-|[([^]]*)]) ([^ "]*|"[^"]*") (-|[0-9]*)

你得到以下输出

1.  66.249.68.6
2.  -
3.  -
4.  14/Jan/2012:06:25:03 -0800
5.  "GET /example.com HTTP/1.1"
6.  200

这篇关于用于访问登录 hive serde 的正则表达式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！