将文本文件解析为表格数据以进行处理 [英] Parsing text file to tabular data for processing
问题描述
当前的问题是使用python以表格形式解析特定数据.一小部分数据如下所示
The problem at hand is to parse a particular data in a tabular form using python.A small part of data is shown below
Statistics indicator:0x222235
number of records = 3
records[0]
value one = 2
value two = 5
blocks = 2
block[0] {
some irrelevant data....
value three = 4 bytes
}
block[1]{
some irrelevant data...
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 4 bytes
}
records[2]
value one = 7
value two = 6
blocks = 2
block[0] {
some irrelevant data....
value three = 3 bytes
}
block[1]{
some irrelevant data...
value three = 4 bytes
}
Statistics indicator:0x135256
number of records = 2
records[0]
value one = 4
value two = 8
blocks = 1
block[0] {
some irrelevant data....
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 3 bytes
}
如图所示，数据具有特定的模式.它在每个特定数据块的开头都有统计指示符.它具有记录数字段以指示数据块具有的记录数.在每个记录中，``值一''和值二``是不同的.但是，取决于每个记录具有的块数，有几个值三"，这些值由块"字段指示. 这里的问题是将以下数据以表格形式排列，并添加与特定记录相对应的所有值三.
As shown , the data has a particular pattern. It has statistics indicator at the start of every particular block of data.It has the number of records field to indicate the number of records the data block has.Within each record the 'value one' and value two' is distinct. However there are several "value three"'s depending on the number of blocks each record has which is indicated by 'blocks'field. The problem here is to arrange the following data in the tabular form adding all the value three's corresponding to the particular record.
决赛桌应该像这样:
一值二值三
2 5 10
3 5 4
7 6 7
4 8 6
3 5 3
我正在考虑的方法是首先找到统计指标" 如果找到统计指标"，我将寻找记录的数量和与每个记录对应的块以遍历这些块，然后将值三加起来对应于相似的值1和值2.
The approach I am thinking is to first find 'Statistics indicator' If I find 'Statistics indicator' I will look for number of records and blocks corresponding to each record to iterate through the blocks and sum value three's corresponding to similar value one and value two.
这是我尝试提取值一，值二和值三的代码.我还没有研究求和值三.
Here is the code which I tried to extract value one value two and value three. I have not delved into summing value three's yet.
import re
import pandas as pd
val_dict = { 'value_one':re.compile(r'value one = (?P<value_one>.*)\n'),
'value_two':re.compile(r'value two = (?P<value_two>.*)\n'),
'value_three':re.compile(r'value three = (?P<value_three>.*)\n')}
def _parse_line(line):
for key, val in val_dict.items():
match = val.search(line)
if match:
return key, match
# if there are no matches
return None, None
def parse_file(filepath):
data = []
with open(filepath, 'r') as file_object:
row = {} # prepare an empty row
for line in file_object:
key, match = _parse_line(line)
# search for keys in the line
if key == 'value_one':
value_one = match.group('value_one')
value_one = int(value_one)
if 'value one' in row: # we always have a full row
data.append(row) # append it to the data liest
row = {} # and reset it
row['value one'] = value_one # we have a match: store the value in row
if key == 'value_two':
value_two = match.group('value_two')
value_two = int(value_two)
if 'value two' in row:
data.append(row)
row = {}
row['value two'] = value_two
if key == 'value_three':
value_three = match.group('value_three')
value_three = int(value_three)
if 'value three' in row:
data.append(row)
row = {}
row['value three'] = value_three
if row != {}: # do not forget the last row
data.append(row)
data = pd.DataFrame(data)
return data
if __name__ == '__main__':
filepath = 'test3.txt'
data = parse_file(filepath)
推荐答案
在这里，也许我们不想使用正则表达式.但是，如果我们愿意，可以将属性名称设置为左边界，并收集所需的数字，也许使用类似以下表达式:
Here, maybe we don't want to use regular expressions. However, if we might want to do so, we can set attribute names as a left boundary, and collect our desired digits, maybe with an expression similar to:
value\s+(one|two|three)\s+=\s+([0-9]+)
然后，可以将其余问题编写成脚本.如果需要，我们还可以在表达式中添加更多边界.
Then, the rest of our problem can be scripted. We can also add more boundaries to our expressions, if that'd be necessary.
# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility
import re
regex = r"value\s+(one|two|three)\s+=\s+([0-9]+)"
test_str = ("Statistics indicator:0x222235\n\n"
"number of records = 3\n\n"
"records[0]\n\n"
"value one = 2\n\n"
"value two = 5\n\n"
"blocks = 2\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"block[1]{\n\n"
"some irrelevant data...\n\n"
"value three = 6 bytes\n\n"
"}\n\n"
"records[1]\n\n"
"value one = 3\n\n"
"value two = 5\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"records[2]\n\n"
"value one = 7\n\n"
"value two = 6\n\n"
"blocks = 2\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 3 bytes\n\n"
"}\n\n"
"block[1]{\n\n"
"some irrelevant data...\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"Statistics indicator:0x135256\n\n"
"number of records = 2\n\n"
"records[0]\n\n"
"value one = 4\n\n"
"value two = 8\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 6 bytes\n\n"
"}\n\n"
"records[1]\n\n"
"value one = 3\n\n"
"value two = 5\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 3 bytes\n\n"
"}")
matches = re.finditer(regex, test_str, re.MULTILINE)
for matchNum, match in enumerate(matches, start=1):
print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))
for groupNum in range(0, len(match.groups())):
groupNum = groupNum + 1
print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))
# Note: for Python 2.7 compatibility, use ur"" to prefix the regex and u"" to prefix the test string and substitution.
演示
const regex = /value\s+(one|two|three)\s+=\s+([0-9]+)/gm;
const str = `Statistics indicator:0x222235
number of records = 3
records[0]
value one = 2
value two = 5
blocks = 2
block[0] {
some irrelevant data....
value three = 4 bytes
}
block[1]{
some irrelevant data...
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 4 bytes
}
records[2]
value one = 7
value two = 6
blocks = 2
block[0] {
some irrelevant data....
value three = 3 bytes
}
block[1]{
some irrelevant data...
value three = 4 bytes
}
Statistics indicator:0x135256
number of records = 2
records[0]
value one = 4
value two = 8
blocks = 1
block[0] {
some irrelevant data....
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 3 bytes
}`;
let m;
while ((m = regex.exec(str)) !== null) {
// This is necessary to avoid infinite loops with zero-width matches
if (m.index === regex.lastIndex) {
regex.lastIndex++;
}
// The result can be accessed through the `m`-variable.
m.forEach((match, groupIndex) => {
console.log(`Found match, group ${groupIndex}: ${match}`);
});
}
如果不希望使用此表达式，则可以在 regex101.com中进行修改或更改
.If this expression wasn't desired, it can be modified or changed in regex101.com.
jex.im 可视化正则表达式:
这篇关于将文本文件解析为表格数据以进行处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
