How to Parse XML in Python

This blog was originally posted to Crawlbase Blog

XML (Extensible Markup Language) is a common format for storing and transferring data between different platforms and systems. As a Python developer working on web services, config files, or data transfer, you need to know how to parse XML files. You can use Python libraries to make XML parsing easy and fast.

This article will cover various ways to parse XML in Python, both built-in libraries and external tools. You’ll learn how to handle XML files of all sizes, convert XML to dictionaries and save parsed data to CSV and JSON. We’ll also look at parsing invalid or malformed XML with more lenient tools.

Let’s dive into the details of how to parse XML in Python.

What is XML?

XML, or Extensible Markup Language, is a data format to store and exchange data between different systems. It’s human-readable and machine-readable, that’s why it’s used in web services, configuration files, and more.

Why XML?

XML is used because it’s a flexible and organized way to represent complex data. Unlike CSV or plain text, XML allows you to create a hierarchy of elements and attributes, so it’s easier to understand and manipulate the data.

Here are a few reasons why XML is preferred:

• Platform independence: XML can be used with any operating system and programming language.
• Scalability: XML files can contain simple and complex data structures.
• Readability: Non-developers can read and understand XML.

What is XML Parsing?

XML parsing is the process of reading and processing an XML document to extract data. In Python, parsing XML allows you to browse XML documents, extract data, and change it as needed. This is especially important when working with APIs or other data exchange systems that use XML as their protocol.

Python has built-in libraries and third-party tools to parse XML data, whether it’s a small config file or a big data source. In the next sections, we’ll see how.

Parsing XML Using Python’s Built-in Libraries

Python has powerful built-in libraries for working with XML data. These libraries allow you to parse XML files, extract what you need, and manipulate the data as required. Two popular libraries in Python for parsing XML are xml.etree.ElementTree and xml.dom.minidom.

Parsing XML with xml.etree.ElementTree

xml.etree.ElementTree is a lightweight library that comes with Python by default. It is an XML parsing library for Python, which was made to easily parse and navigate through XML files.

For example, this is how you might use ElementTree to parse an XML string:

import xml.etree.ElementTree as ET

# Example XML data
xml_data = """
<catalog>
    <product>
        <name>Wireless Mouse</name>
        <price>29.99</price>
        <category>Electronics</category>
    </product>
    <product>
        <name>Office Chair</name>
        <price>89.99</price>
        <category>Furniture</category>
    </product>
</catalog>
"""

# Parse the XML data
root = ET.fromstring(xml_data)

# Access and print data
for product in root.findall('product'):
    name = product.find('name').text
    price = product.find('price').text
    category = product.find('category').text
    print(f"Product Name: {name}, Price: ${price}, Category: {category}")

In this example, we start by importing the ElementTree module. We can get the root element of an XML by parsing the XML string using fromstring() method. We will then use the findall(), and find() methods to search for a particular tag or extract text from inside tags.

Parsing XML with xml.dom.minidom

xml.dom.minidom is another built-in library that uses the Document Object Model (DOM) to parse and manipulate XML. It is more powerful and flexible, but it can be less easy to use than ElementTree in some cases.

The same XML data can be parsed with minidom as follows:

from xml.dom.minidom import parseString

# Example XML data
xml_data = """
<catalog>
    <product>
        <name>Wireless Mouse</name>
        <price>29.99</price>
        <category>Electronics</category>
    </product>
    <product>
        <name>Office Chair</name>
        <price>89.99</price>
        <category>Furniture</category>
    </product>
</catalog>
"""

# Parse the XML data
dom = parseString(xml_data)

# Access and print data
products = dom.getElementsByTagName('product')
for product in products:
    name = product.getElementsByTagName('name')[0].childNodes[0].nodeValue
    price = product.getElementsByTagName('price')[0].childNodes[0].nodeValue
    category = product.getElementsByTagName('category')[0].childNodes[0].nodeValue
    print(f"Product Name: {name}, Price: ${price}, Category: {category}")

In this example, parseString() is used to load the XML into a DOM object. We then use getElementsByTagName() to find the product, name, price, and category elements, and childNodes[0].nodeValue to extract the text. While minidom offers detailed control, it’s generally less efficient than ElementTree for simple tasks.

Working with External XML Parsing Libraries

You might want to stick with the built-in libraries for simple tasks when you are working with XML data in Python. On the other hand, for more complex requirements or better handling incorrect XML, you might choose another library to have more features and flexibility. In this part, we will discuss two famous external XML parsing libraries: lxml and BeautifulSoup.

Parsing XML with lxml

You can also use lxml which is a very nice library when you are working with XML and HTML documents. It has good support for XPath and XSLT hence could be a powerful XML processor.

To get started with lxml, you'll need to install it. You can do this via pip:

pip install lxml

Here's an example of how to use lxml to parse XML data:

from lxml import etree

# Example XML data
xml_data = """
<catalog>
    <product>
        <name>Wireless Mouse</name>
        <price>29.99</price>
        <category>Electronics</category>
    </product>
    <product>
        <name>Office Chair</name>
        <price>89.99</price>
        <category>Furniture</category>
    </product>
</catalog>
"""

# Parse the XML data
root = etree.fromstring(xml_data)

# Access and print data
for product in root.xpath('//product'):
    name = product.find('name').text
    price = product.find('price').text
    category = product.find('category').text
    print(f"Product Name: {name}, Price: ${price}, Category: {category}")

In this example, we use lxml's etree module to parse XML. With the xpath method, you can write powerful queries so that you can extract precisely from this messy XML structure.

How to Handle Malformed XML with BeautifulSoup

BeautifulSoup is often used for parsing HTML, but it can also handle malformed XML gracefully. This makes it a good choice for dealing with XML documents that may not be well-formed.

To use BeautifulSoup for XML parsing, install the library along with a parser like lxml:

pip install beautifulsoup4 lxml

Here’s an example of using BeautifulSoup to parse XML:

from bs4 import BeautifulSoup

# Example XML data with a malformed element
xml_data = """
<catalog>
    <product>
        <name>Wireless Mouse</name>
        <price>29.99</price>
        <category>Electronics</category>
    </product>
    <product>
        <name>Office Chair</name>
        <price>89.99</price>
        <category>Furniture</category>
    </product>
    <!-- Missing closing tag for catalog -->
"""

# Parse the XML data
soup = BeautifulSoup(xml_data, 'lxml-xml')

# Access and print data
for product in soup.find_all('product'):
    name = product.find('name').get_text()
    price = product.find('price').get_text()
    category = product.find('category').get_text()
    print(f"Product Name: {name}, Price: ${price}, Category: {category}")

In this case, BeautifulSoup helps parse incomplete or broken XML documents. It is especially useful when you run into XML that deviates from the common practices of writing XML.

In the next part, we will look at how to transform XML data into Python dictionaries for better manipulation.

How to Convert XML to Dictionary in Python

Working with XML data may be difficult, if the requirement is to edit or extract some elements of it. A common workaround is to convert the XML into a Python dictionary. A dictionary is used for storing and retrieving data as key-value pairs making us deal with data more easily. Let’s explore two popular libraries for converting XML into a Python dictionary: xmltodict and untangle.

Using xmltodict

xmltodict is a simple library that can convert XML data to a dictionary in a few lines of code. It simplifies and speeds up the processing of XML data.

To get started, you’ll need to install the library using pip:

pip install xmltodict

Here’s an example of how to use xmltodict to convert XML into a dictionary:

import xmltodict

# Example XML data
xml_data = """
<store>
    <item>
        <name>Notebook</name>
        <price>5.99</price>
        <quantity>100</quantity>
    </item>
    <item>
        <name>Pencil</name>
        <price>0.99</price>
        <quantity>500</quantity>
    </item>
</store>
"""

# Convert XML to a dictionary
data_dict = xmltodict.parse(xml_data)

# Access and print data
for item in data_dict['store']['item']:
    name = item['name']
    price = item['price']
    quantity = item['quantity']
    print(f"Item: {name}, Price: ${price}, Quantity: {quantity}")

In this example, xmltodict.parse() converts the XML data into a Python dictionary, allowing you to work with it as if it were a standard dictionary. This makes it much easier to retrieve and manipulate data from XML.

Using untangle

Another great library for parsing XML into Python objects is untangle. Unlike xmltodict, which converts XML into a dictionary, untangle turns the XML into Python objects that you can easily access through attributes.

First, install the library using pip:

pip install untangle

Here’s an example of how to use untangle:

import untangle

# Example XML data
xml_data = """
<store>
    <item>
        <name>Notebook</name>
        <price>5.99</price>
        <quantity>100</quantity>
    </item>
    <item>
        <name>Pencil</name>
        <price>0.99</price>
        <quantity>500</quantity>
    </item>
</store>
"""

# Parse XML into Python objects
data = untangle.parse(xml_data)

# Access and print data
for item in data.store.item:
    name = item.name.cdata
    price = item.price.cdata
    quantity = item.quantity.cdata
    print(f"Item: {name}, Price: ${price}, Quantity: {quantity}")

In this example, untangle converts the XML structure into Python objects. Each XML tag becomes an attribute of the object, and you can easily access the content using cdata (character data).

Next, we’ll look at how to save the parsed XML data into different formats like CSV or JSON for further use.

How to Save Parsed XML Data

After parsing XML data, it is generally saved in more familiar forms such as CSV or JSON. This allows you to save, exchange, and analyze data in a more user-friendly manner for the majority of apps. In this part, we'll look at two ways to save parsed XML data: exporting it to CSV with pandas and saving it as JSON.

Exporting to CSV with pandas

CSV (Comma-Separated Values) files are commonly used to store tabular data. Python's pandas package makes it simple to save parsed XML data to a CSV file.To get started, make sure pandas is installed:

pip install pandas

Here’s an example of how to convert XML data into a CSV file using pandas:

import xml.etree.ElementTree as ET
import pandas as pd

# Example XML data
xml_data = """
<store>
    <item>
        <name>Notebook</name>
        <price>5.99</price>
        <quantity>100</quantity>
    </item>
    <item>
        <name>Pencil</name>
        <price>0.99</price>
        <quantity>500</quantity>
    </item>
</store>
"""

# Parse XML
root = ET.fromstring(xml_data)

# Extract data and create a list of dictionaries
data = []
for item in root.findall('item'):
    name = item.find('name').text
    price = item.find('price').text
    quantity = item.find('quantity').text
    data.append({'Name': name, 'Price': price, 'Quantity': quantity})

# Convert list of dictionaries to a pandas DataFrame
df = pd.DataFrame(data)

# Save DataFrame to a CSV file
df.to_csv('store_items.csv', index=False)

print("Data has been saved to store_items.csv")

In this example, we use xml.etree.ElementTree to parse the XML data, and then we extract relevant information (like name, price, and quantity) into a list of dictionaries. pandas is then used to create a DataFrame and save the data to a CSV file.

Saving Data to JSON

JSON (JavaScript Object Notation) is a lightweight data format used in web applications and APIs. Python has a built-in module called json that can convert parsed XML to JSON.

Here’s how to convert XML to JSON and save to a file:

import xmltodict
import json

# Example XML data
xml_data = """
<store>
    <item>
        <name>Notebook</name>
        <price>5.99</price>
        <quantity>100</quantity>
    </item>
    <item>
        <name>Pencil</name>
        <price>0.99</price>
        <quantity>500</quantity>
    </item>
</store>
"""

# Convert XML to a dictionary using xmltodict
data_dict = xmltodict.parse(xml_data)

# Convert dictionary to JSON and save to a file
with open('store_items.json', 'w') as json_file:
    json.dump(data_dict, json_file, indent=4)

print("Data has been saved to store_items.json")

In this example, we use xmltodict to convert the XML to a dictionary and then the json module to convert that dictionary to JSON. The JSON is saved to a file called store_items.json.

Next, we will cover how to handle large XML files.

Handling Large XML Files

Loading the entire file into memory can be slow and inefficient when dealing with large XML files. To address this, it’s better to use memory-friendly strategies that allow for processing the XML in smaller chunks. One effective way is to parse the XML file incrementally, reducing memory usage and speeding up processing time for large datasets.

Stream Parsing with iterparse

Stream parsing is an efficient technique for handling large XML files by processing them in chunks, instead of reading the whole file at once. Python’s iterparse method allows you to process XML data as it is being parsed, making it ideal for XML files that are too large to fit into memory.

Here’s how iterparse works:

1. Parse events: With iterparse, you can define events like 'start' or 'end' to trigger actions when an XML element starts or ends. This gives you control over how and when each part of the XML is processed.
2. Memory management: After processing each element, you can clear it from memory to minimize memory usage, which is crucial when handling large XML files.

Example:

import xml.etree.ElementTree as ET

# Stream parse the XML file
for event, element in ET.iterparse('large_file.xml', events=('end',)):
    if element.tag == 'product':
        # Extract product data
        name = element.find('name').text
        category = element.find('category').text
        price = element.find('price').text
        print(f"Product: {name}, Category: {category}, Price: {price}")

        # Clear the processed element from memory
        element.clear()

This example processes each <product> element individually and then destroys the object to keep memory usage down. This method is very helpful when dealing with XML files with thousands or millions of elements.

Final Thoughts

Python has multiple flexible tools for dealing with XML, e.g. the built-in xml.etree.ElementTree to more advanced external third-party packages, like lxml and BeautifulSoup. Python can take care of anything you throw at it, such as simple parsing, converting XML to dictionaries, or handling large and malformed files.

With the right tool, you can parse XML quickly and format it into either CSV or JSON.Using the methods discussed in the blog, you can easily handle XML parsing in Python.

For more tutorials like these, follow our blog. If you have any questions or feedback, our support team is here to help you.

Frequently Asked Questions (FAQs)

Q. Is Python good for parsing XML?

Yes, Python is excellent for parsing XML. It has built-in libraries like xml.etree.ElementTree and xml.dom.minidom make XML parsing easy and efficient. Third-party libraries like lxml and BeautifulSoup, on the other hand, have more advanced features are designed specifically to deal with complex or malformed XML data.

Q. What is the best Python library for XML parsing?

Which is the best library depends on one's needs. ElementTree is often enough for simple stuff. lxml or BeautifulSoup plugins are more suitable if you need to handle poorly-formed XML, boost processing speed, etc.

Q. How can I convert XML to a dictionary in Python?

Use libraries like xmltodict or untangle to Convert XML to Python dictionary easily. The libraries provide an easy way to convert XML data into Python dictionaries, allowing you to interact with and use your data easily.