将包含多个XML文件的单个大文件读入C#中的多个xml记录 [英] Read single large file containing multiple XML files into multiple xml records in C#

问题描述

我有一个文件，其中实际上包含了多个相同格式的XML文件，因此该文件本身不是有效的XML；例如:

I have a single file that effectively contains multiple XML files of the same format, so the file is not itself valid XML; for instance:

<?xml version='1.0' encoding='UTF-8'?>
<Proposal xmlns="a namespace">
    <ASubnode>Text</ASubNode>
    <LotsOfOtherNodes />
</Proposal>
<?xml version='1.0' encoding='UTF-8'?>
<Proposal xmlns="a namespace">
    <ASubnode>Text</ASubNode>
    <LotsOfOtherNodes />
</Proposal>
....

我想一次处理所有投标"节点；例如:

I would like to process all the Proposal nodes, one at a time; for example:

foreach (var proposal in file)
    do something

我不能使用XmlReader，因为它在到达中间XML声明节点时会引发异常.我可以将整个文件读取为一个字符串，然后使用Split方法，但是这些文件的大小为千兆字节，因此作为一种选择并不是特别吸引人.似乎我可以一次读取一行文件，通过正则表达式搜索适当的节点，但是文件的格式不是如上所述的，每行一个节点，而是包含很长的多个节点行，以及节点文本中的随机换行符.

I cannot use XmlReader because it throws an exception upon reaching the intermediate XML declaration nodes. I could possibly read the entire file into a string and then use the Split method, but these files are Gigabytes in size, so that is not particularly attractive as an option. It might seem that I could read the file a line at a time, searching for the appropriate nodes via a regular expression, but the files are not line-formatted as above with one node per line, but rather contain very long lines of multiple nodes, and random line breaks in node text.

是否有一种无需手工制作文本解析器即可实现的方法?

Is there a method of achieving this without hand-crafting a text parser?

推荐答案

由于xml文档的标题是相同的，因此您可以逐行阅读文本而无需实际解析xml:

You can read the text line by line without actually parsing the xml, since the header of an xml document is the same:

IEnumerable<XDocument> GetDocuments(Stream bulkStream)
{
    var reader = new StreamReader(bulkStream);
    var sb = new StringBuilder();   
    var firstLine = reader.ReadLine();
    string line = firstLine;    
    while(line != null)
    {
        sb.Clear();
        sb.Append(firstLine);
        while((line = reader.ReadLine()) != null && line != firstLine)
        {
            sb.Append(line);
        }

        yield return XDocument.Parse(sb.ToString());
    }
}

更新:即使声明可以在一行之间开始，也可以执行以下操作:

UPDATE: Following will work even if the declarations can start in-between of a line:

IEnumerable<XDocument> GetDocuments(Stream bulkStream)
{
    const string decl = @"<?xml version='1.0' encoding='UTF-8'?>";
    var sb = new StringBuilder();   

    bool start = true;
    foreach(var line in GetLines(bulkStream).Where(l => !string.IsNullOrWhiteSpace(l)))
    {
        if (start)
        {
            if (line == decl)
                start = false;
            sb.AppendLine(line);
        }
        else
        {
            if (line == decl)
            {
                sb.ToString().Dump();
                yield return XDocument.Parse(sb.ToString());

                sb.Clear();
                start = true;
                sb.AppendLine(line);
            }
            else
                sb.AppendLine(line);
        }
    }

    sb.ToString().Dump();
    yield return XDocument.Parse(sb.ToString());
}

IEnumerable<string> GetLines(Stream bulkStream)
{
    const string decl = @"<?xml version='1.0' encoding='UTF-8'?>";
    var reader = new StreamReader(bulkStream);
    string line;
    while((line = reader.ReadLine()) != null)
    {
        if (line.Contains(decl))
        {
            var declIndex = line.IndexOf(decl);
            yield return line.Substring(0, declIndex);
            yield return decl;
            yield return line.Substring(declIndex + decl.Length);
        }
        else
        {
            yield return line;
        }
    }
}

这篇关于将包含多个XML文件的单个大文件读入C#中的多个xml记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！