Dealing with huge xml-like files containing illegal characters

t3mplar — Sat, 26 Jun 2021 18:09:37 +0000

Sometimes you have to deal with files that look pretty like xml. But htey are not, because contain a lot of illegal characters. Such files usually made by just concatenating strings and not verifying angainst any schema. When these files are comparably small, it's not a big deal to use regex to replace all those characters, but for really big files it's not very convinient.

The idea was to read file element by element and if next part is not a valid xml, then I could clean all illegal characters and use well-formed xml for my needs.

public IEnumerable<XElement> GetElement(string filePath, string elementName)
{
    using var reader = XmlReader.Create(filePath, new XmlReaderSettings { CheckCharacters = false });
    reader.MoveToContent();

    while (reader.Read())
    {
        if (reader.NodeType == XmlNodeType.Element && reader.Name == elementName)
        {
            var str = reader.ReadOuterXml();
            XNode node;

            try
            {
                node = XElement.Parse(str);
            }
            catch (XmlException)
            {
                var pattern = @"&#x((10?|[2-F])FFF[EF]|FDD[0-9A-F]|[19][0-9A-F]|7F|8[0-46-9A-F]|0?[1-8BCEF]);";
                var regex = new Regex(pattern, RegexOptions.IgnoreCase);
                var fixedStr = regex.Replace(str, string.Empty);

                node = XElement.Parse(fixedStr);
            }

            if (node is XElement el)
            {
                yield return el;
            }
        }
    }
}

Important note:
To skip checking for invalid characters, you should pass to XmlReader settings XmlReaderSettings { CheckCharacters = false } so it can omit checks and give me possibility to cleanup input string.

DEV Community: t3mplar

Dealing with huge xml-like files containing illegal characters