使用 Xpath 处理较大 XML 文件的最佳方法是什么? [英] What is the best way to use Xpath for processing larger XML files?

问题描述

我有一个要求，我必须使用大型 XML(4 GB 文件)来查找其中的值.基本上我必须编写大约 30 个不同的 Xpath 并将值存储在列表中.当我尝试解析 XML 时，它会引发内存错误.我曾尝试将 lxml 和 ElementTree 与开始和结束事件一起使用，但仍然没有运气，处理时间太长，我的 Pycharm/Jupyter 笔记本引发了内存错误.

I have a requirement where I have to use an large XML (4 GB file) for finding values in it. Basically I have to write around 30 different Xpath and store the values in a list. When I try to parse an XML, it throws memory error. I have tried using lxml and ElementTree with start and end events, still no luck the processing time is too high and my Pycharm/Jupyter notebook throws me memory error.

有没有更好的方法来做到这一点?尽管此实现不限于任何编程语言，但我更喜欢 Python，因为它是我的右手.提前致谢.

Is there a better way to do it? Even though this implementation is not restricted to any programming language, I prefer Python because its my right hand. Thanks in advance.

例如搜索:如果我想要类别正在烹饪的年份的值.然后我使用 ./bookstore[@category=cooking]/book/year.所以值为 2005

Eg of a search: If I want the value of year where category is cooking. Then I use ./bookstore[@category=cooking]/book/year. So the value is 2005

同样，我必须根据我的业务需求找到我的标签值.在我的要求中，XML 并不像下面的例子那样简单.

Similarly I have to find the values of my tags based on my business requirements. In my requirement the XML is not simple as the below example.

<?xml version="1.0" encoding="UTF-8"?>

<bookstore>

<book category="cooking">
  <title lang="en">Everyday Italian</title>
  <author>Giada De Laurentiis</author>
  <year>2005</year>
  <price>30.00</price>
</book>

<book category="children">
  <title lang="en">Harry Potter</title>
  <author>J K. Rowling</author>
  <year>2005</year>
  <price>29.99</price>
</book>

<book category="web">
  <title lang="en">XQuery Kick Start</title>
  <author>James McGovern</author>
  <author>Per Bothner</author>
  <author>Kurt Cagle</author>
  <author>James Linn</author>
  <author>Vaidyanathan Nagarajan</author>
  <year>2003</year>
  <price>49.99</price>
</book>

</bookstore>

推荐答案

我不知道这是否能满足您的需求.先试试.如果有问题，我们可以继续沟通.

I don't know if this will meet your needs. Try it first. If there is a problem, we can continue to communicate.

from simplified_scrapy import SimplifiedDoc,req,utils
def getBook(lines,category,year):
  html = "".join(lines)
  book = SimplifiedDoc(html).book
  if book.category==category and book.year.text==year:
    return book.category,book.title.text,book.authors.text,book.year.text,book.price.text
lst = []
with open('bookstore.xml', 'r') as file: # bookstore.xml is your xml file path
  lines = []
  flag = False
  for line in file:
    if flag or line.find('<book ')>=0:
      flag = True
      lines.append(line)
    if line.find('</book>')>=0:
      b = getBook(lines,'web','2003')
      if b: 
        lst.append(b)
        break # If you only want the first one, add a break
      flag = False
      lines = []
print (lst)

结果:

[('web', 'XQuery Kick Start', ['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan'], '2003', '49.99')]

这篇关于使用 Xpath 处理较大 XML 文件的最佳方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！