Xml – How to split an XML document into thirds (or, even better, n pieces)

xml

I would like to use a language that I am familiar with – Java, C#, Ruby, PHP, C/C++, although examples in any language or pseudocode are more than welcome.

What is the best way of splitting a large XML document into smaller sections that are still valid XML? For my purposes, I need to split them into roughly thirds or fourths, but for the sake of providing examples, splitting them into n components would be good.

Best Solution

Parsing XML documents using DOM doesn't scale.

This Groovy-script is using StAX (Streaming API for XML) to split an XML document between the top-level elements (that shares the same QName as the first child of the root-document). It's pretty fast, handles arbitrary large documents and is very useful when you want to split a large batch-file into smaller pieces.

Requires Groovy on Java 6 or a StAX API and implementation such as Woodstox in the CLASSPATH

import javax.xml.stream.*

pieces = 5
input = "input.xml"
output = "output_%04d.xml"
eventFactory = XMLEventFactory.newInstance()
fileNumber = elementCount = 0

def createEventReader() {
    reader = XMLInputFactory.newInstance().createXMLEventReader(new FileInputStream(input))
    start = reader.next()
    root = reader.nextTag()
    firstChild = reader.nextTag()
    return reader
}

def createNextEventWriter () {
    println "Writing to '${filename = String.format(output, ++fileNumber)}'"
    writer = XMLOutputFactory.newInstance().createXMLEventWriter(new FileOutputStream(filename), start.characterEncodingScheme)
    writer.add(start)
    writer.add(root)
    return writer
}

elements = createEventReader().findAll { it.startElement && it.name == firstChild.name }.size()
println "Splitting ${elements} <${firstChild.name.localPart}> elements into ${pieces} pieces"
chunkSize = elements / pieces
writer = createNextEventWriter()
writer.add(firstChild)
createEventReader().each { 
    if (it.startElement && it.name == firstChild.name) {
        if (++elementCount > chunkSize) {
            writer.add(eventFactory.createEndDocument())
            writer.flush()
            writer = createNextEventWriter()
            elementCount = 0
        }
    }
    writer.add(it)
}
writer.flush()

Related Solutions

C# – How to build XML in C#

It depends on the scenario. XmlSerializer is certainly one way and has the advantage of mapping directly to an object model. In .NET 3.5, XDocument, etc. are also very friendly. If the size is very large, then XmlWriter is your friend.

For an XDocument example:

Console.WriteLine(
    new XElement("Foo",
        new XAttribute("Bar", "some & value"),
        new XElement("Nested", "data")));

Or the same with XmlDocument:

XmlDocument doc = new XmlDocument();
XmlElement el = (XmlElement)doc.AppendChild(doc.CreateElement("Foo"));
el.SetAttribute("Bar", "some & value");
el.AppendChild(doc.CreateElement("Nested")).InnerText = "data";
Console.WriteLine(doc.OuterXml);

If you are writing a large stream of data, then any of the DOM approaches (such as XmlDocument/XDocument, etc.) will quickly take a lot of memory. So if you are writing a 100 MB XML file from CSV, you might consider XmlWriter; this is more primitive (a write-once firehose), but very efficient (imagine a big loop here):

XmlWriter writer = XmlWriter.Create(Console.Out);
writer.WriteStartElement("Foo");
writer.WriteAttributeString("Bar", "Some & value");
writer.WriteElementString("Nested", "data");
writer.WriteEndElement();

Finally, via XmlSerializer:

[Serializable]
public class Foo
{
    [XmlAttribute]
    public string Bar { get; set; }
    public string Nested { get; set; }
}
...
Foo foo = new Foo
{
    Bar = "some & value",
    Nested = "data"
};
new XmlSerializer(typeof(Foo)).Serialize(Console.Out, foo);

This is a nice model for mapping to classes, etc.; however, it might be overkill if you are doing something simple (or if the desired XML doesn't really have a direct correlation to the object model). Another issue with XmlSerializer is that it doesn't like to serialize immutable types : everything must have a public getter and setter (unless you do it all yourself by implementing IXmlSerializable, in which case you haven't gained much by using XmlSerializer).

C# – How to Deserialize XML document

How about you just save the xml to a file, and use xsd to generate C# classes?

Write the file to disk (I named it foo.xml)
Generate the xsd: xsd foo.xml
Generate the C#: xsd foo.xsd /classes

Et voila - and C# code file that should be able to read the data via XmlSerializer:

    XmlSerializer ser = new XmlSerializer(typeof(Cars));
    Cars cars;
    using (XmlReader reader = XmlReader.Create(path))
    {
        cars = (Cars) ser.Deserialize(reader);
    }

(include the generated foo.cs in the project)

Best Solution

Related Solutions

C# – How to build XML in C#

C# – How to Deserialize XML document

Related Question