删除重复的行而不进行排序 [英] Remove duplicate lines without sorting
问题描述
我在Python中有一个实用程序脚本:
I have a utility script in Python:
#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
if line in unique_lines:
duplicate_lines.append(line)
else:
unique_lines.append(line)
sys.stdout.write(line)
# optionally do something with duplicate_lines
这个简单的功能(uniq
,无需先排序,保持稳定的顺序)必须作为一个简单的UNIX实用程序提供,不是吗?也许是管道中的过滤器组合?
This simple functionality (uniq
without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?
问的原因:在无法从任何地方执行Python的系统上需要此功能.
Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.
推荐答案
The UNIX Bash Scripting blog suggests:
awk '!x[$0]++'
此命令告诉awk要打印哪些行.变量$0
包含一行的全部内容,方括号可用于数组访问.因此,对于文件的每一行,如果先前未设置该节点的内容(!
),则数组x
的节点将递增,并打印该行.
This command is telling awk which lines to print. The variable $0
holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x
is incremented and the line printed if the content of that node was not (!
) previously set.
这篇关于删除重复的行而不进行排序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!