写大的JSON文件,避免内存不足问题的最佳办法 [英] Best way of writing Big JSON files avoiding OutOfMemory issues
问题描述
首先,请注意,今天是我第一天 GSON
。我想写使用 GSON
库中的JSON文件。我有成千上万的 JsonObjects
内的的ArrayList
。当JSON文件写入,它看起来应该与此类似。
First, please note today is my first day with GSON
. I am trying to write a Json file using GSON
library. I have thousands of JsonObjects
inside an ArrayList
. When the Json file is written, it should look like similar to this.
[
{
"hash_index": "00102x05h06l0aj0dw",
"body": "Who's signing up for Obamacare?",
"_type": "ArticleItem",
"title": "Who's signing up for Obamacare? - Jan. 13, 2014",
"source": "money.cnn.com",
"primary_key": 0,
"last_crawl_date": "2014-01-14",
"url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
},
{
"hash_index": "00102x05h06l0aj0dw0iz0kn0l@0t#0",
"body": "Who's signing up for Obamacare?",
"_type": "ArticleItem",
"title": "Who's signing up for Obamacare? - Jan. 13, 2014",
"source": "money.cnn.com",
"primary_key": 1,
"last_crawl_date": "2014-01-14",
"url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
}
]
现在,我写的JSON通过使用下面的code。
Right now, I write the JSOn by using the below code.
private void writeNewJsonFile() throws IOException
{
System.out.println("Starting to write the JSON File");
//Add everything into a JSONArray
JsonArray jsonArrayNew = new JsonArray();
for(int i=0;i<jsonObjectHolder.size();i++)
{
System.out.println("inside array");
jsonArrayNew.add(jsonObjectHolder.get(i));
}
//Write it to the File
/* File file= new File("items_Articles_4_1.json");
FileWriter fw = new FileWriter(file);;
fw.write(jsonArrayNew.toString());
fw.flush();
fw.close();*/
System.out.println("outside array");
ByteArrayInputStream input = new ByteArrayInputStream(jsonArrayNew.toString().getBytes());
Long contentLength = Long.valueOf(jsonArrayNew.toString().getBytes().length);
ObjectMetadata metaData = new ObjectMetadata();
metaData.setContentLength(contentLength);
s3.putObject(outputBucket,outputFile,input,metaData);
}
在这里,我将在 JsonArray
到字符串
并做写作。我有一个担心,这将很快崩溃,大的Json数组,并给我的 OutOfMemoryException异常
。就像我读使用GSON JSON的文件,一部分一部分,是有什么办法,我可以写JSON文件逐件什么的,可避免 OutOfMemoryException异常
问题?
Here I am converting the JsonArray
into a String
and do the writing. I have a fear that this will soon crash with Big Json arrays and give me the OutOfMemoryException
. Just like I read the Json files part by part using GSON, is there is any way I can write the Json file piece by piece or something, which can avoid the OutOfMemoryException
issues?
推荐答案
我使用的下一个code:
I am using next code:
WriteJsonArrayByParts<Cache> write = new WriteJsonArrayByParts<Cache>(fileNameTest, " ");
write.writeStart();
for(Cache cache : listOfObjects()) {
write.writeObject(cache, Cache.class);
}
write.writeEnd();
write.close();
...
public static class WriteJsonArrayByParts<T> {
Gson gson = new Gson();
JsonWriter writer;
public WriteJsonArrayByParts(String fileNameWithPath, String indent) throws Exception {
OutputStream os = new FileOutputStream(fileNameWithPath, false);
BufferedOutputStream osb = new BufferedOutputStream(os, 8 * 1024);
writer = new JsonWriter(new OutputStreamWriter(osb, StringUtil.UTF_8));
writer.setIndent(indent);
}
public void writeStart() throws IOException {
writer.beginArray();
}
@SuppressWarnings("unchecked")
public void writeObject(T t, Class<?> resultClass) throws IOException {
((TypeAdapter<Object>) gson.getAdapter(resultClass)).write(writer, t);
}
public void writeEnd() throws IOException {
writer.endArray();
}
public void close() throws IOException {
writer.close();
}
}
这篇关于写大的JSON文件,避免内存不足问题的最佳办法的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!