unix工具从文件中删除重复的行 [英] unix tool to remove duplicate lines from a file
问题描述
diff
。我想删除重复项,以便我可以很容易地比较它们。基本上,像 sort -u
但没有排序。 有没有可以做到这一点的unix命令行工具?
概要
uniq [OPTION] ... [INPUT [OUTPUT]]
描述
从INPUT(或标准输入)中删除除了一个连续的相同行之外的所有输入,写入OUTPUT(或标准输出)。
或者,如果要删除不相邻的重复行,那么perl的这个片段将会执行:
while(<>){
print $ _ if(!$ seen {$ _});
$ seen {$ _} = 1;
}
I have a tool that generates tests and predicts the output. The idea is that if I have a failure I can compare the prediction to the actual output and see where they diverged. The problem is the actual output contains some lines twice, which confuses diff
. I want to remove the duplicates, so that I can compare them easily. Basically, something like sort -u
but without the sorting.
Is there any unix commandline tool that can do this?
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).
Or, if you want to remove non-adjacent duplicate lines as well, this fragment of perl will do it:
while(<>) {
print $_ if (!$seen{$_});
$seen{$_}=1;
}
这篇关于unix工具从文件中删除重复的行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!