批量合并大型CSV文件并删除重复的标头 [英] Batch combine large CSV files and remove repeating header

问题描述

我正在尝试将多个具有相同字段的大型csv文件组合在一起.我能做的最好的事情是:

I am trying to combine several large csv files with identical fields. The best I've been able to do is:

@echo off
setlocal
set first=1
for %%F in (*.csv) do (
  if defined first (
    type "%%F" >> new.csv
    set "first="
  ) else (
    more +1 "%%F" >> new.csv
  )
) 

这在较小的文件上效果很好，但是当文件较大时，more命令会卡在每个屏幕转储的末尾.

This works great on small files, but the more command gets stuck at the end of each screen dump when the files are larger.

尽我所能，将其写入new.csv时，我无法获得更多的性能来直接运行整个文件-

Try as I might, I haven't been able to get more to run straight through the entire file when writing it to new.csv - can anyone help?

推荐答案

@ECHO OFF
SET first=y
SET newfile=new.csv
for %%F in (*.csv) do IF NOT %%F==%newfile% (
  if defined first (
    COPY /y "%%F" %newfile% >nul
    set "first="
  ) else (
    FOR /f "skip=1delims=" %%i IN (%%F) DO >> %newfile% ECHO %%i
  )
) 

编辑
要在CSV中支持空行，您可以将内部FOR替换为:
对于/f代码"，跳过= 1令牌= 1 * delims ="在('findstr/n"^" %% F')中执行%% i>>％newfile％echo.%% j

这篇关于批量合并大型CSV文件并删除重复的标头的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！