如何删除与AWK部分重复线路？ [英] How to delete partial duplicate lines with AWK?

问题描述

我有这几样重复行，只有最后一个字段是不同的文件：

I have files with these kind of duplicate lines, where only the last field is different:

OST,0202000070,01-AUG-09,002735,6,0,0202000068,4520688,-1,0,0,0,0,0,55
ONE,0208076826,01-AUG-09,002332,316,3481.055935,0204330827,29150,200,0,0,0,0,0,5
ONE,0208076826,01-AUG-09,002332,316,3481.055935,0204330827,29150,200,0,0,0,0,0,55
OST,0202000068,01-AUG-09,003019,6,0,0202000071,4520690,-1,0,0,0,0,0,55

我需要删除该行的第一次出现，并留下第二个。

I need to remove the first occurrence of the line and leave the second one.

我试过：

awk '!x[$0]++ {getline; print $0}' file.csv

但如预期，因为它也除去未重复行它不工作。

but it's not working as intended, as it's also removing non duplicate lines.

推荐答案

如果您近期重复总是相邻的，你可以比较的previous条目，并避免产生潜在的巨大关联数组。

If your near-duplicates are always adjacent, you can just compare to the previous entry and avoid creating a potentially huge associative array.

#!/bin/awk -f
{
    s = substr($0, 0, match($0, /,[^,]*$/))
    if (s != prev) {
        print prev0
    }
    prev = s
    prev0 = $0
} 
END {
    print $0
}

修改：改变了脚本，以便它打印一组近乎重复的最后一个（没有​​ TAC 需要）

Edit: Changed the script so it prints the last one in a group of near-duplicates (no tac needed).

这篇关于如何删除与AWK部分重复线路？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！