将大型CSV文件转换为JSON [英] Convert large CSV files to JSON
问题描述
我不介意如果这是一个单独的程序，用Excel，在NodeJS或在一个网络应用程序。
I don't mind if this is done with a separate program, with Excel, in NodeJS or in a web app.
这是完全相同的问题，这里：
It's exactly the same problem as described here:
似乎OP没有得到工作的答案（仍然接受它？） 。我尝试使用它，但似乎不能让它工作。
It seems that the OP didn't get that answer to work (yet accepted it anyway?). I've tried working with it but can't seem to get it to work either.
简而言之：我使用一个〜50,000行CSV，我想将其转换为JSON。我已经尝试过几乎每个在线的csv到jsonwebapp，所有崩溃与这个大的数据集。
In short: I'm working with a ~50,000 row CSV and I want to convert it to JSON. I've tried just about every online "csv to json" webapp out there, all crash with this large of a dataset.
我试过很多 Node CSV到JSON模块，但是，它们都崩溃了。 csvtojson 模块似乎很有希望，但我得到这个错误：致命错误：JS分配失败 - 进程内存不足。
I've tried many Node CSV to JSON modules but, again, they all crash. The csvtojson module seemed promising, but I got this error: FATAL ERROR: JS Allocation failed - process out of memory.
我能做些什么来获得这些数据的可用格式？如上所述，我不介意它是一个应用程序，在Excel，webapp或Node模块中工作，只要我得到一个 .JSON 文件或
What on earth can I do to get this data in a useable format? As above, I don't mind if it's an application, something that works within Excel, a webapp or a Node module, so long as I either get a .JSON file or an object that I can work with within Node.
任何想法？
推荐答案
您在上面提到了csvtojson模块，这是一个我正在维护的开源项目。
You mentioned csvtojson module above and that is an open source project which I am maintaining.
对不起，它没有为你工作，几个月前解决的一个bug。
我还为您的场景在README中添加了一些额外的行。请查看在命令中处理大CSV文件第行。
I am sorry it did not work out for you and it was caused by a bug solved several months ago. I also added some extra lines in README for your scenario. Please check out Process Big CSV File in Command Line.
请确保您拥有最新的csvtojson发行版。 （目前为0.2.2）
Please make sure you have the latest csvtojson release. (Currently it is 0.2.2)
您可以通过运行
npm install -g csvtojson
在您安装最新的csvtojson ，您只需运行：
csvtojson [path to bigcsvdata] > converted.json
这将从csvfile流式传输数据。或者，如果你想从另一个应用程序流数据：
This streams data from the csvfile. Or if you want to stream data from another application:
cat [path to bigcsvdata] | csvtojson > converted.json
他们将输出相同的东西。
They will output the same thing.
我手动测试了一个csv文件超过300万条记录，它的工作没有问题。
I have manually tested it with a csv file over 3 million records and it works without an issue.
我相信你只需要一个简单的工具。 lib的目的是像这样释放压力。如果下次遇到任何问题，请与我们联系，以便及时解决。
I believe you just need a simple tool. The purpose of the lib is to relief stress like this. Please do let me know if you meet any problems next time so I could solve it in time.
这篇关于将大型CSV文件转换为JSON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
