如何提高Write-Progress的性能? [英] How to improve the performance of Write-Progress?
问题描述
我正在编写一个脚本，它从另一个平台获取一个输出文件(遗憾的是它不会产生 CSV 输出，而是每条记录大约 7 行)，抓取具有我感兴趣的值的行(使用select-string) 然后扫描 MatchInfo 数组，提取准确的文本并构建一个数组，完成后导出到 CSV.
I'm writing a script that takes an output file from another platform (that sadly doesn't produce CSV output, instead it's around 7 lines per record), grabbing the lines that have the values I'm interested in (using select-string) and then scanning the MatchInfo array, extracting the exact text and building an array as I go, to export to CSV when finished.
我的问题是原始文件有大约 94000 行文本，而 matchinfo 对象中仍有大约 23500 条记录，所以需要一段时间，尤其是构建数组，所以我想我会抛出一个 Write-Progress 但这样做的开销非常可怕，它增加了 x8 的运行时间，而不是没有进度条.
My problem is that the original file has around 94000 lines of text, and the matchinfo object still has around 23500 records in it, so it takes a while, especially building the array, so I thought I'd throw in a Write-Progress but the overhead in doing so is quite horrific, it increases the elapsed time x8 vs not having the progress bar.
这是原始文件中的一个示例条目:
Here's an example entry from the original file:
CREATE TRANCODE MPF OF TXOLID
AGENDA = T4XCLCSHINAG
,ANY_SC_LIST = NONE ,EVERY_SC_LIST = NONE
,SECURITY_CATEGORY = NONE ,FUNCTION = 14
,TRANCODE_VALUE = "MPF"
,TRANCODE_FUNCTION_MNEMONIC = NONE
,INSTALLATION_DATA = NONE
;
现在，对于其中的每一个，我只关心 AGENDA 和 TRANCODE_VALUE 的值，因此使用 Get-Content，然后我使用 Select-String 作为我所知道的最有效的方法来过滤掉文件中的其余行:
Now, for each of these, I only care about the values of AGENDA and TRANCODE_VALUE, so having read the file in using Get-Content, I then use Select-String as the most efficient way I know to filter out the rest of the lines in the file:
rv Start,Filtered,count,CSV
Write-Host "Reading Mainframe Extract File"
$Start = gc K:\TRANCODES.txt
Write-Host ("Read Complete : " + $Start.Count + " records found")
Write-Host "Filtering records for AGENDA/TRANCODE information"
$Filtered = $Start|Select-String -Pattern "AGENDA","TRANCODE_VALUE"
Write-Host ([String]($Filtered.Count/2) + " AGENDA/TRANCODE pairs found")
这给我留下了一个 Microsoft.PowerShell.Commands.MatchInfo 类型的对象，内容如下:
This leaves me with an object of type Microsoft.PowerShell.Commands.MatchInfo with contents like:
AGENDA = T4XCLCSHINAG
,TRANCODE_VALUE = "MPF"
AGENDA = T4XCLCSHINAG
,TRANCODE_VALUE = "MP"
现在 Select-String 只花了大约 9 秒，所以真的不需要那里的进度条.
Now that Select-String only took around 9 seconds, so no real need for a progress bar there.
但是，下一步，获取实际值(在 = 之后)并放入一个数组需要 30 多秒，所以我认为 Write-Progress 是对用户有帮助，至少表明某些事情确实在发生，但是，进度条的添加严重延长了经过的时间，请参阅 Measure-Command 的以下输出:
However, the next step, grabbing the actual values (after the =) and putting in an array takes over 30 seconds, so I figured a Write-Progress is helpful to the user and at least shows that something is actually happening, but, the addition of the progress bar seriously extends the elapsed time, see the following output from Measure-Command:
Measure-Command{$Filtered|foreach {If ($_.ToString() -Match 'AGENDA'){$obj = $null;
$obj = New-Object System.Object;
$obj | Add-Member -type NoteProperty -name AGENDA -Value $_.ToString().SubString(27)}
If ($_.ToString() -Match 'TRANCODE_VALUE'){$obj | Add-Member -type NoteProperty -name TRANCODE -Value ($_.ToString().SubString(28)).Replace('"','');
$CSV += $obj;
$obj = $null}
<#$count++
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Processed " + $count + " of " + $Filtered.Count + " records") `
-Id 1 `
-PercentComplete ([int]($count/$Filtered.Count *100))#>
}}
TotalSeconds : 32.7902523
所以这是 717.2308630680085 条记录/秒
So that's 717.2308630680085 records/sec
Measure-Command{$Filtered|foreach {If ($_.ToString() -Match 'AGENDA'){$obj = $null;
$obj = New-Object System.Object;
$obj | Add-Member -type NoteProperty -name AGENDA -Value $_.ToString().SubString(27)}
If ($_.ToString() -Match 'TRANCODE_VALUE'){$obj | Add-Member -type NoteProperty -name TRANCODE -Value ($_.ToString().SubString(28)).Replace('"','');
$CSV += $obj;
$obj = $null}
$count++
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Processed " + $count + " of " + $Filtered.Count + " records") `
-Id 1 `
-PercentComplete ([int]($count/$Filtered.Count *100))
}}
TotalSeconds : 261.3469632
现在只有微不足道的 89.98660799693897 条记录/秒
Now only a paltry 89.98660799693897 records/sec
有什么提高效率的想法吗?
Any ideas how to improve the efficiency?
完整脚本如下:
rv Start,Filtered,count,CSV
Write-Host "Reading Mainframe Extract File"
$Start = gc K:\TRANCODES.txt
Write-Host ("Read Complete : " + $Start.Count + " records found")
Write-Host "Filtering records for AGENDA/TRANCODE information"
$Filtered = $Start|Select-String -Pattern "AGENDA","TRANCODE_VALUE"
Write-Host ([String]($Filtered.Count/2) + " AGENDA/TRANCODE pairs found")
Write-Host "Building table from the filter results"
[int]$count = 0
$CSV = @()
$Filtered|foreach {If ($_.ToString() -Match 'AGENDA'){$obj = $null;
$obj = New-Object System.Object;
$obj | Add-Member -type NoteProperty -name AGENDA -Value $_.ToString().SubString(27)}
If ($_.ToString() -Match 'TRANCODE_VALUE'){$obj | Add-Member -type NoteProperty -name TRANCODE -Value ($_.ToString().SubString(28)).Replace('"','');
$CSV += $obj;
$obj = $null}
$count++
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Processed " + $count + " of " + $Filtered.Count + " records") `
-Id 1 `
-PercentComplete ([int]($count/$Filtered.Count *100))
}
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Table built : " + $CSV.Count + " rows created") `
-Id 1 `
-Completed
Write-Host ("Table built : " + $CSV.Count + " rows created")
Write-Host "Sorting and Exporting table to CSV file"
$CSV|Select TRANCODE,AGENDA|Sort TRANCODE |Export-CSV -notype K:\TRANCODES.CSV
以下是注释掉 write-progress 的脚本输出:
Here's output from script with the write-progress commented out:
Reading Mainframe Extract File
Read Complete : 94082 records found
Filtering records for AGENDA/TRANCODE information
11759 AGENDA/TRANCODE pairs found
Building table from the filter results
Table built : 11759 rows created
Sorting and Exporting table to CSV file
TotalSeconds : 75.2279182
我采用了@RomanKuzmin 答案的修改版本，因此相应的代码部分现在如下所示:
I've adopted a modified version of the answer from @RomanKuzmin, so the appropriate code section now looks like:
Write-Host "Building table from the filter results"
[int]$count = 0
$CSV = @()
$sw = [System.Diagnostics.Stopwatch]::StartNew()
$Filtered|foreach {If ($_.ToString() -Match 'AGENDA'){$obj = $null;
$obj = New-Object System.Object;
$obj | Add-Member -type NoteProperty -name AGENDA -Value $_.ToString().SubString(27)}
If ($_.ToString() -Match 'TRANCODE_VALUE'){$obj | Add-Member -type NoteProperty -name TRANCODE -Value ($_.ToString().SubString(28)).Replace('"','');
$CSV += $obj;
$obj = $null}
$count++
If ($sw.Elapsed.TotalMilliseconds -ge 500) {
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Processed " + $count + " of " + $Filtered.Count + " records") `
-Id 1 `
-PercentComplete ([int]($count/$Filtered.Count *100));
$sw.Reset();
$sw.Start()}
}
Write-Progress `
-Activity "Building table of values from filter results" `
-Status ("Table built : " + $CSV.Count + " rows created") `
-Id 1 `
-Completed
并通过 Measure-Command 运行整个脚本，在没有 write-progress 和修改后的 write-progress 的情况下，运行时间为 75.2279182 秒> 使用@RomanKuzmin 的建议，76.525382 秒 - 一点也不差！！:-)
And running the entire script through Measure-Command gives elapsed time of 75.2279182 seconds with no write-progress and with the modified write-progress using @RomanKuzmin suggestion, 76.525382 seconds - not bad at all!! :-)
推荐答案
在这种情况下，如果经常调用进度，我会使用这种方法
In such cases when progress is called too often I use this approach
# fast even with Write-Progress
$sw = [System.Diagnostics.Stopwatch]::StartNew()
for($e = 0; $e -lt 1mb; ++$e) {
if ($sw.Elapsed.TotalMilliseconds -ge 500) {
Write-Progress -Activity Test -Status "Done $e"
$sw.Reset(); $sw.Start()
}
}
# very slow due to Write-Progress
for($e = 0; $e -lt 1mb; ++$e) {
Write-Progress -Activity Test -Status "Done $e"
}
这里是关于 连接....
这篇关于如何提高Write-Progress的性能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
