如何从文件中删除重复的行 [英] How to remove duplicate lines from a file

问题描述

我有一个生成测试并预测输出的工具.这个想法是，如果我失败了，我可以将预测与实际输出进行比较，看看它们之间的差异.问题是实际输出包含两次某些行，这混淆了diff.我想删除重复项，以便可以轻松比较它们.基本上是类似sort -u的东西，但是没有排序.

I have a tool that generates tests and predicts the output. The idea is that if I have a failure I can compare the prediction to the actual output and see where they diverged. The problem is the actual output contains some lines twice, which confuses diff. I want to remove the duplicates, so that I can compare them easily. Basically, something like sort -u but without the sorting.

是否有可以执行此操作的UNIX命令行工具?

Is there any unix command line tool that can do this?

推荐答案

uniq( 1)

简介

uniq [OPTION] ... [INPUT [OUTPUT]]

uniq [OPTION]... [INPUT [OUTPUT]]

说明

从INPUT(或标准输入)中丢弃连续的同一行中的所有行，但只丢弃其中之一，写入OUTPUT(或标准输出)中.

Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

或者，如果您也想删除不相邻的重复行，则此perl片段可以做到这一点:

Or, if you want to remove non-adjacent duplicate lines as well, this fragment of perl will do it:

while(<>) {
    print $_ if (!$seen{$_});
    $seen{$_}=1;
}

这篇关于如何从文件中删除重复的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！