删除重复行而不排序 [英] Remove duplicate lines without sorting

问题描述

我有一个 Python 实用程序脚本:

I have a utility script in Python:

#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
  if line in unique_lines:
    duplicate_lines.append(line)
  else:
    unique_lines.append(line)
    sys.stdout.write(line)
# optionally do something with duplicate_lines

这个简单的功能(uniq 无需先排序，稳定排序)必须作为一个简单的 UNIX 实用程序可用，不是吗?也许是管道中过滤器的组合?

This simple functionality (uniq without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?

询问原因:在我无法从任何地方执行 Python 的系统上需要此功能.

Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.

推荐答案

UNIX Bash 脚本博客 建议:

The UNIX Bash Scripting blog suggests:

awk '!x[$0]++'

这个命令告诉 awk 要打印哪些行.变量 $0 保存一行的全部内容，方括号是数组访问.因此，对于文件的每一行，数组 x 的节点都会递增，并且如果该节点的内容不是 (!) 先前设置的，则会打印该行.

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.

这篇关于删除重复行而不排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！