过滤大型CSV文件时出现内存异常 [英] Memory exception while filtering large CSV files

问题描述

在运行此代码时获取内存异常.有没有一种方法可以一次过滤一个文件，并在处理每个文件后写入输出并追加.似乎下面的代码将所有内容加载到内存中.

getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.

$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
    ForEach-Object { Import-Csv $_.FullName } |
    Where-Object { $_.machine_type -eq 'workstations' } |
    Export-Csv $outputFile -NoType

推荐答案

也许可以一个接一个地导出和过滤文件，然后将结果附加到输出文件中，如下所示:

May be can you export and filter your files one by one and append result into your output file like this :

$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"

Remove-Item $outputFile -Force -ErrorAction SilentlyContinue

Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }

这篇关于过滤大型CSV文件时出现内存异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！