JSON Powershell 内存问题 [英] JSON Powershell memory issue

问题描述

我使用命令读取一个 JSON 文件，这一切都很好，直到文件变大.

I use a command to read a JSON file, this all works perfectly, until the file becomes large.

我目前有一个大约 1.5GB 的 JSON 文件.我使用 Powershell 使用以下命令读取文件:

I currently have a JSON file of about 1.5GB. I read the file using Powershell using the following command:

get-content -Path C:\TEMP\largefile.json | out-string | ConvertFrom-Json

它返回以下错误:

out-string : Exception of type 'System.OutOfMemoryException' was thrown.
+ ... oices = get-content -Path C:\TEMP\largefile.json | out-string | Conve ...
+                                                        ~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Out-String], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand

我已经增加了内存，如下所示:

I've increased the memory as shown here:

get-item wsman:localhost\Shell\MaxMemoryPerShellMB


WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   MaxMemoryPerShellMB                            8096

关于如何处理这个有什么想法吗?

Any ideas on how to process this?

编辑(根据评论添加):

Edit (additions based on comments):

当我删除输出字符串时，出现此错误:

When I remove the out-string I get this error:

ConvertFrom-Json : Exception of type 'System.OutOfMemoryException' was thrown.
    + ... oices = get-content -Path C:\TEMP\largefile.json | ConvertFrom-Json ...
    +                                                        ~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Out-String], OutOfMemoryException
    + FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand

我拥有的 Powershell 版本是:5.1.17763.1490

The Powershell version that I have is: 5.1.17763.1490

该文件包含多个关于 PDF 文件的列.这些文件通过 API 导出到 JSON 中，因此它包含文件元数据，例如所有者和创建时间，还包含列 Body 中的实际 PDF 文件，稍后将解码为实际的 PDF 文件.结构如下:

The file contains multiple columns regarding PDF files. These files are exported via an API into a JSON so it contains the file metadata such as owner and when it was created but also the actual PDF file in the column Body which later will be decoded to an actual PDF file. The structure is as followed:

[{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
]

推荐答案

感谢提供详细信息.
对于这个问题，我会尝试分别转换每一行并通过您的流程进行流式传输:

Thank for the details.
For this issue I would try to convert each line separately and stream that through your process:

Get-Content C:\TEMP\largefile.json | ForEach-Object {
    $_ = $_.Trim().TrimStart('[').TrimEnd(']')
    if ($_) { $_ | ConvertFrom-Json }
}

正如已经建议的那样，如果这些内存问题不会出现在 PowerShell 核心.如果可能，我建议您也尝试一下.

As already suggested, I wouldn't be surprised if these memory issues wouldn't appear in PowerShell core. if possible, I recommend you to also give that a try.

这篇关于JSON Powershell 内存问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！