如何在R中读取大型(〜20 GB)xml文件? [英] How to read large (~20 GB) xml file in R?

问题描述

我想从大型xml文件(20 GB)中读取数据并进行操作.我厌倦了使用"xmlParse()"，但是在加载之前它给了我内存问题.有什么有效的方法可以做到这一点吗?

I want to read data from large xml file (20 GB) and manipulate them. I tired to use "xmlParse()" but it gave me memory issue before loading. Is there any efficient way to do this?

我的数据转储看起来像这样

My data dump looks like this,

<tags>                                                                                                    
    <row Id="106929" TagName="moto-360" Count="1"/>
    <row Id="106930" TagName="n1ql" Count="1"/>
    <row Id="106931" TagName="fable" Count="1" ExcerptPostId="25824355" WikiPostId="25824354"/>
    <row Id="106932" TagName="deeplearning4j" Count="1"/>
    <row Id="106933" TagName="pystache" Count="1"/>
    <row Id="106934" TagName="jitter" Count="1"/>
    <row Id="106935" TagName="klein-mvc" Count="1"/>
</tags>

推荐答案

在XML包中，xmlEventParse函数实现SAX(读取XML并调用函数处理程序).如果您的XML很简单(在一个根元素内重复元素)，则可以使用branches参数为每个元素定义函数.

In XML package the xmlEventParse function implements SAX (reading XML and calling your function handlers). If your XML is simple enough (repeating elements inside one root element), you can use branches parameter to define function(s) for every element.

示例:

MedlineCitation = function(x, ...) {
  #This is a "branch" function
  #x is a XML node - everything inside element <MedlineCitation>
  # find element <ArticleTitle> inside and print it:
  ns <- getNodeSet(x,path = "//ArticleTitle")
  value <- xmlValue(ns[[1]])
  print(value)
}

调用XML解析:

xmlEventParse(
  file = "http://www.nlm.nih.gov/databases/dtd/medsamp2015.xml", 
  handlers = NULL, 
  branches = list(MedlineCitation = MedlineCitation)
)

封闭解决方案:

就像马丁·摩根一样，存储特定于xml-node- values-with-rs-xmleventparse :

branchFunction <- function() {
  store <- new.env() 
  func <- function(x, ...) {
    ns <- getNodeSet(x, path = "//ArticleTitle")
    value <- xmlValue(ns[[1]])
    print(value)
    # if storing something ... 
    # store[[some_key]] <- some_value
  }
  getStore <- function() { as.list(store) }
  list(MedlineCitation = func, getStore=getStore)
}

myfunctions <- branchFunction()

xmlEventParse(
  file = "medsamp2015.xml", 
  handlers = NULL, 
  branches = myfunctions
)

#to see what is inside
myfunctions$getStore()

这篇关于如何在R中读取大型(〜20 GB)xml文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！