如何删除重复的线路中的Unix文件？ [英] How can I delete duplicate lines in a file in Unix?

问题描述

有没有办法删除一个文件重复行的Unix？

我可以用做排序-u 和 uniq的命令。但我想用 SED 或 AWK 。
这可能吗？

解决方案

 的awk'！看到[$ 0] ++'file.txt的
 

看到是一个关联数组来，awk将通过文​​件的每一行。如果行是不是数组中那么看到[$ 0] 将评估为假。在！是一个逻辑NOT运算符和将反转假为真。 awk将打印其中前pression计算结果为真行。在 ++ 递增看到让看到[$ 0] == 1 后的第一时间线被发现，然后看到[$ 0] == 2 ，等等。结果
AWK评估一切，但 0 和，（空字符串）为true。如果重复线路被放置在看到然后！看到[$ 0] 将评估为false，该行不会写入输出

Is there way to delete duplicate lines in a file in Unix?

I can do it with sort -u and uniq commands. but I want to use sed or awk. Is that possible?

解决方案

awk '!seen[$0]++' file.txt

seen is an associative-array that Awk will pass every line of the file to. If a line isn't in the array then seen[$0] will evaluate to false. The ! is a logical NOT operator and will invert the false to true. Awk will print the lines where the expression evaluates to true. The ++ increments seen so that seen[$0] == 1 after the first time a line is found and then seen[$0] == 2, and so on.
Awk evaluates everything but 0 and "" (empty string) to true. If a duplicate line is placed in seen then !seen[$0] will evaluate to false and the line will not be written to the output.

这篇关于如何删除重复的线路中的Unix文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！