The Day I Automated XML Field Checking with Python

It all started when I was given the task of checking several XML files for missing fields. The team needed to ensure that all mandatory fields were present in these files before we could proceed with our next steps. Sounds simple enough, right? Well, not quite.

I opened the first XML file, scanned through the properties, manually looked for the required fields, and ticked the boxes as I went. As you might expect, it got tiring very quickly. After just a couple of minutes in one file, my eyes were glazing over, and I had no real confidence that I hadn’t missed something crucial. I mean, XMLs can be so finicky, and a single missing field could cause major problems down the line.

I had this gnawing feeling of dread, knowing I still had a bunch of files to go through. And, of course, accuracy was critical—one overlooked missing field could spell disaster. So after taking a few deep breaths and a moment to think, I decided there had to be a better way to tackle this.

The Epiphany: Automation to the Rescue

Being a programmer, I had an idea: why not write a script to do this monotonous work for me? Instead of manually checking every single field, I could automate it and guarantee accuracy while saving my sanity in the process. It was time to harness the power of Python.

The concept was simple:

• I had a list of required fields stored in a JSON file, which made the script highly reusable and adaptable. By using this approach, the script can easily process other XML files, even those with different structures. You simply need to update the JSON file with the required fields for any new XML format, allowing the script to automatically adjust to different XML schemas without modification.
• I needed to write a Python script that would go through each XML file, check if any of the required fields were missing, and then output a summary.

This way, I could easily identify how many times a field was missing in each file, how many properties were present, and get a clear report—no more endless manual checks, no more mistakes. Here’s how I approached it.

Writing the Utility Script

First things first, I needed to load the list of required fields. These were stored in a JSON file under the key required_fields, so I wrote a function to read this file:

import os
import json
import xml.etree.ElementTree as ET

def load_required_fields(json_file_path):
    with open(json_file_path, 'r') as file:
        data = json.load(file)
        return data.get("required_fields", [])

Then came the real magic. I wrote a function to parse each XML file, loop through its properties, and check for the presence of each required field:

def check_missing_fields(file_path, required_fields):
    # Load the XML file
    tree = ET.parse(file_path)
    root = tree.getroot()

    # Initialize variables to store counts and track missing fields
    total_properties = 0
    missing_fields_counts = {field: 0 for field in required_fields}

    # Loop through each property to check for missing fields
    for property in root.findall('.//property'):
        total_properties += 1
        for field in required_fields:
            # Use the find() method to look for direct children of the property element
            element = property.find(f'./{field}')
            # Check if the field is completely missing (not present)
            if element is None:
                missing_fields_counts[field] += 1

    # Print the results
    print('-----------------------------------------')
    print(f'File: {os.path.basename(file_path)}')
    print(f'Total number of properties: {total_properties}')
    print('Number of properties missing each field:')
    for field, count in missing_fields_counts.items():
        print(f'  {field}: {count} properties')
    print('-----------------------------------------')

This function loaded an XML file, counted the number of properties, and kept track of how many properties were missing each required field. The function printed out a report showing the results for each file processed.

Finally, I put everything together in the main() function. It would iterate over all the XML files in a specified directory and run the field-checking function on each of them:

def main():
    # Directory containing XML files
    xml_dir = 'xmls'
    json_file_path = 'required_fields.json'

    # Load required fields from JSON file
    required_fields = load_required_fields(json_file_path)

    # Iterate over each file in the xmls directory
    for file_name in os.listdir(xml_dir):
        if file_name.endswith('.xml'):
            file_path = os.path.join(xml_dir, file_name)
            check_missing_fields(file_path, required_fields)

if __name__ == "__main__":
    main()

After running the process, you will receive a summary of the results similar to this:

File: properties.xml
Total number of properties: 4170
Number of properties missing each field:
  Title: 0 properties
  Unit_Number: 0 properties
  Type: 0 properties
  Bedrooms: 0 properties
  Bathrooms: 0 properties
  Project: 0 properties
  Price: 0 properties
  VAT: 0 properties
  Status: 10 properties
  Area: 0 properties
  Location: 100 properties
  Latitude: 30 properties
  Longitude: 0 properties
  Apartment_Floor: 0 properties
  Block: 0 properties
  Phase: 0 properties
  Construction_Stage: 0 properties
  Plot_Size: 0 properties
  Yard: 120 properties
  Description: 0 properties
  gallery: 27 properties

The Results: Sanity Saved

Once I had everything in place, I ran the script on my directory of XML files. The output was exactly what I needed: a concise summary showing me how many properties in each file were missing which fields, and the total count of properties in each XML.

Instead of spending hours manually checking each file, I got my answer in a matter of seconds. The script caught several missing fields that I might have overlooked if I had continued down the manual route.

Lessons Learned

1. Automation is a lifesaver: Whenever you’re faced with repetitive tasks, think about how you can automate them. Not only will it save you time, but it will also reduce the risk of human error.
2. Accuracy matters: In situations like these, accuracy is paramount. A simple script like the one I wrote can ensure that you don’t overlook anything, which is especially important when dealing with critical data.
3. Leverage your programming skills: Sometimes, we get caught up in doing things manually, even when we have the skills to make our lives easier. Take a moment to step back and ask yourself, “Is there a more efficient way to do this?”

In the end, what started as a tiresome, error-prone task turned into a rewarding experience. Now, whenever I get tasks that feel tedious or prone to mistakes, I remind myself of the power of scripting and automation. I wonder how many other tasks I can streamline next…

You can quickly get started with this automation by cloning the XML Checker repository I’ve created. This will give you everything you need, including the script and the example files. From there, you’ll be able to run the automation yourself, customize it to fit your needs or extend its functionality even further.

Enjoy!