如果存在，Gulp 删除重复项 [英] Gulp remove duplicates if exists

问题描述

是否可以从源中删除同名文件?例如，假设我有以下文件夹结构

Is it possible to remove files with same name from source? For example, let's say I have the following folder structure

a
 ---file1.txt
 ---file2.txt
 ---file3.txt
b
 ---file1.txt

当我选择源中的两个文件夹时，我只希望目标文件夹中的文件不重复.在上面的示例中，结果将是

When I select both folder in source I want in destination folder only file that aren't duplicates. In example above result would be

 result
   ---file2.txt
   ---file3.txt

可选，如果我能以某种方式复制并在单独的文件夹中进行过滤和写入，那就太好了.重复，我的意思是按名称明确重复，文件内容并不重要.

Optional, it would be great if I could duplicates somehow filter and write in separate folder. By duplicates, I mean explicitly duplicates by name, file content is not important.

推荐答案

我花了一段时间才到达那里，但试试这个:

It took me awhile to get there but try this:

var gulp = require('gulp');
var fs = require('fs');
var path = require('path');
var flatten = require('gulp-flatten');
var filter =  require('gulp-filter');

var folders = ['a', 'b', 'c'];  // I just hard-coded your folders here

    // this function is called by filter for each file in the above folders
    // it should return false if the file is a duplicate, i.e., occurs
    // in at least two folders
function isUnique(file) {

  console.dir(file.history[0]);  // just for fun
  var baseName = file.history[0].split(path.sep);
  baseName = baseName[baseName.length - 1];

     // var fileParents = '././';
  var fileParents = '.' + path.sep + '.' + path.sep;
  var count = 0;

  folders.forEach(function (folder) {
     if (fs.existsSync(fileParents + folder + path.sep + baseName)) count++;
       // could quit forEach when count >= 2 if there were a lot of folders/files
       // but there is no way to break out of a forEach
  });

  if (count >= 2) {  // the file is a duplicate          
    fs.unlinkSync(file.history[0]); // remove from 'Result' directory
    return false;
 }
 else return true;
}

gulp.task('default', ['clump'], function () {
     // create a filter to remove duplicates
  const f = filter(function (file) { return isUnique(file); }, {restore: true, passthrough: false} );

  const stream = gulp.src('./result/*.txt')
   .pipe(f);  // actually do the filtering here

  f.restore.pipe(gulp.dest('duplicates'));  // new stream with the removed duplicates
  return stream;
});

     // 'clump' runs first 
     // gathers all files into result directory
gulp.task('clump', function () {
  return gulp.src('./**/*.txt')    
   .pipe(flatten())  // because the original folder structure in not wanted
   .pipe(gulp.dest('result'));
});

使用gulp"运行它.默认任务会先触发'clump'任务.

Run it with 'gulp'. The default task will trigger the 'clump' task first.

由于您的 OP 不要求保留任何特定版本的重复文件(例如最新版本或其他任何版本)，因此我在这里并不担心.如果在结果"文件夹中，您想要每个版本的重复文件，例如 file1.txt(来自一个文件夹的版本)和 file1.txt(来自另一个文件夹)，但显然必须重命名为可以在丛的任务.

Since your OP didn't require that any particular version of duplicated files be kept - like the newest or whatever - I haven't worried about that here. If in the 'Result' folder you want each version of a duplicated file, such as file1.txt (version from one folder) and file1.txt (from another folder) but obviously must be renamed to something that could be done in the 'clump' task.

让我知道这是否适合你.

Let me know if this works for you.

这篇关于如果存在，Gulp 删除重复项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！