unix工具从文件中删除重复的行 [英] unix tool to remove duplicate lines from a file

问题描述

我有一个工具可以生成测试并预测输出。这个想法是，如果我失败了，我可以将预测与实际产出进行比较，看看它们在哪里分歧。问题是实际的输出包含一些行两次，这混淆了 diff 。我想删除重复项，以便我可以很容易地比较它们。基本上，像 sort -u 但没有排序。

有没有可以做到这一点的unix命令行工具？

解决方案

uniq（1）

概要

uniq [OPTION] ... [INPUT [OUTPUT]]

描述

从INPUT（或标准输入）中删除除了一个连续的相同行之外的所有输入，写入OUTPUT（或标准输出）。

或者，如果要删除不相邻的重复行，那么perl的这个片段将会执行：

  while（<>）{
 print $ _ if（！$ seen {$ _}）; 
 $ seen {$ _} = 1; 
} 
  

I have a tool that generates tests and predicts the output. The idea is that if I have a failure I can compare the prediction to the actual output and see where they diverged. The problem is the actual output contains some lines twice, which confuses diff. I want to remove the duplicates, so that I can compare them easily. Basically, something like sort -u but without the sorting.

Is there any unix commandline tool that can do this?

解决方案

uniq(1)

SYNOPSIS

uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION

Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Or, if you want to remove non-adjacent duplicate lines as well, this fragment of perl will do it:

while(<>) {
    print $_ if (!$seen{$_});
    $seen{$_}=1;
}

这篇关于unix工具从文件中删除重复的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！