Gulp删除重复项，如果存在 [英] Gulp remove duplicates if exists

问题描述

是否可以从源文件中删除具有相同名称的文件？例如，假设我有以下文件夹结构：

  a 
 --- file1.txt 
 --- file2.txt 
 --- file3.txt 
b 
 --- file1.txt 
  

当我在源文件夹中选择两个文件夹时，我只想在目标文件夹中存在不重复的文件。在上面的例子中，结果是：

  result 
 --- file2.txt 
 --- file3。 txt 
  

可选的，如果我可以重复过滤并在单独的文件夹中写入，
通过重复，我的意思是显式重复的名称，文件内容并不重要。

解决方案

我花了一段时间那么试试这个：

  var gulp = require（'gulp'）; 
 var fs = require（'fs'）; 
 var path = require（'path'）; 
 var flatten = require（'gulp-flatten'）; 
 var filter = require（'gulp-filter'）; 
 
 var folders = ['a'，'b'，'c']; //我在这里硬编码你的文件夹
 
 //这个函数被过滤器调用，用于上述文件夹中的每个文件
 //如果文件是重复的，它应该返回false，即在至少两个文件夹中出现
 // 
函数isUnique（file）{
 
 console.dir（file.history [0]）; //只是为了好玩
 var baseName = file.history [0] .split（path.sep）; 
 baseName = baseName [baseName.length  -  1]; 
 
 // var fileParents ='././'; 
 var fileParents ='。'+ path.sep +'。'+ path.sep; 
 var count = 0; 
 
 folders.forEach（函数（文件夹）{
 if（fs.existsSync（fileParents +文件夹+ path.sep + baseName））count ++; 
 //可以退出forEach when如果有大量的文件夹/文件
 // //但没有办法突破forEach 
}，那么计数> = 2; 
 
 if（count> = 2）{//文件是重复的
 fs.unlinkSync（file.history [0]）; //从'结果'目录中移除
 return false; 
} 
 else返回true; 
 
 $ b $ gulp.task（'default'，['clump']，function（）{
 //创建一个过滤器来删除重复
 const f = filter（function（file）{return isUnique（file）;}，{restore：true，passthrough：false}）; 
 
 const stream = gulp.src（'./ result / *。txt' ）
 .pipe（f）; //实际上在这里做过滤
 
 f.restore.pipe（gulp.dest（'duplicates'））; //带删除重复项的新流
返回流; 
}）; 
 
 //'clump'首先运行
 //将所有文件收集到结果目录中
 gulp.task（'clump'，function（）{
 return gulp。 src（'./**/*。txt'）
 .pipe（flatten（））//因为不需要原始文件夹结构
 .pipe（gulp.dest（'result'）） ; 
}）; 
  

使用'gulp'运行它。首先，默认任务会触发'clump'任务。

由于您的OP不需要保留任何特定版本的重复文件 - 像最新的或其他的 - 我并不担心这件事。如果在'结果'文件夹中需要每个版本的重复文件，例如file1.txt（来自一个文件夹的版本）和file1.txt（来自另一个文件夹），但显然必须重命名为可以在' clump'任务。

让我知道这是否适合您。

Is it possible to remove files with same name from source? For example, let's say I have the following folder structure

a
 ---file1.txt
 ---file2.txt
 ---file3.txt
b
 ---file1.txt

When I select both folder in source I want in destination folder only file that aren't duplicates. In example above result would be

 result
   ---file2.txt
   ---file3.txt

Optional, it would be great if I could duplicates somehow filter and write in separate folder. By duplicates, I mean explicitly duplicates by name, file content is not important.

解决方案

It took me awhile to get there but try this:

var gulp = require('gulp');
var fs = require('fs');
var path = require('path');
var flatten = require('gulp-flatten');
var filter =  require('gulp-filter');

var folders = ['a', 'b', 'c'];  // I just hard-coded your folders here

    // this function is called by filter for each file in the above folders
    // it should return false if the file is a duplicate, i.e., occurs
    // in at least two folders
function isUnique(file) {

  console.dir(file.history[0]);  // just for fun
  var baseName = file.history[0].split(path.sep);
  baseName = baseName[baseName.length - 1];

     // var fileParents = '././';
  var fileParents = '.' + path.sep + '.' + path.sep;
  var count = 0;

  folders.forEach(function (folder) {
     if (fs.existsSync(fileParents + folder + path.sep + baseName)) count++;
       // could quit forEach when count >= 2 if there were a lot of folders/files
       // but there is no way to break out of a forEach
  });

  if (count >= 2) {  // the file is a duplicate          
    fs.unlinkSync(file.history[0]); // remove from 'Result' directory
    return false;
 }
 else return true;
}

gulp.task('default', ['clump'], function () {
     // create a filter to remove duplicates
  const f = filter(function (file) { return isUnique(file); }, {restore: true, passthrough: false} );

  const stream = gulp.src('./result/*.txt')
   .pipe(f);  // actually do the filtering here

  f.restore.pipe(gulp.dest('duplicates'));  // new stream with the removed duplicates
  return stream;
});

     // 'clump' runs first 
     // gathers all files into result directory
gulp.task('clump', function () {
  return gulp.src('./**/*.txt')    
   .pipe(flatten())  // because the original folder structure in not wanted
   .pipe(gulp.dest('result'));
});

Run it with 'gulp'. The default task will trigger the 'clump' task first.

Since your OP didn't require that any particular version of duplicated files be kept - like the newest or whatever - I haven't worried about that here. If in the 'Result' folder you want each version of a duplicated file, such as file1.txt (version from one folder) and file1.txt (from another folder) but obviously must be renamed to something that could be done in the 'clump' task.

Let me know if this works for you.

这篇关于Gulp删除重复项，如果存在的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！