In the previous episode, we taught our native parser based on libclang to find system headers. It stopped panicking and finally read all the code without errors.
But when we ran the documentation generation and started verifying completeness metrics, we discovered a disaster. Hundreds of methods were missing from our generated classes. The parser had simply "lost" them along the way.
The Core Problem: Out-Of-Line Methods
In C++, it is considered good practice to move the implementation of methods outside the class declaration. Usually, the implementation goes into separate .cpp files, and in the case of template classes, it remains in the header file but is written below the class itself. Both variants are called Out-Of-Line definitions:
template<class T> class MyClass {
void foo();
};
template<class T> void MyClass<T>::foo() { ... }
Good old Doxygen forgave such things: it parsed the code as text and, by the MyClass:: prefix, easily guessed where to put the method.
How libclang "Loses" Children
But libclang builds a strict semantic Abstract Syntax Tree (AST). In its architecture, a class declaration is a single node. And an Out-Of-Line method is a completely independent top-level node (Top-level declaration), hanging somewhere in the global scope of the translation unit.
libclang fundamentally does not nest this method inside the class cursor during tree traversal. As a result, our AST traversal script saw the class, saw that there were no method nodes inside it, and happily generated an empty documentation page.
Surgical Splicing
To restore justice, we had to write our own tree reconstruction algorithm.
We introduced a separate parser pass: we traversed all "free-hanging" functions in the global scope and asked Clang for their semantic parent (semantic_parent). If it turned out that a function semantically belonged to some class, we took that node and "manually" stitched it inside the appropriate class in our Intermediate Representation (ProjectCatalog).
We literally assembled the fragmented code like a puzzle.
Nameless Conversions
But the surprises didn't end there. The second pain point was type casting operators for dependent templates (for example, cv::Ptr<T>::operator T*()).
We discovered that for such nodes (with the CONVERSION_FUNCTION type), libclang outputs an empty string as a name—cursor.spelling returned "". Meaning the method is there, but it has no name!
We had to write a dirty but working hack: we parsed the node's return type and manually synthesized the string name of the operator from it so that the documentation would look adequate.
Conclusion
This "garbage collection" and manual AST tree splicing restored our method completeness (Recall) to 97.4%.
We achieved almost perfect data extraction. It was time to apply this parser in real combat for a large corporate client. From the very beginning, we knew about the business's main strict requirement: the output of our modern parser had to completely match their old documentation generator. Now it was time to put this into practice. How we aligned the file structure and routing—read about it in the next episode.
Originally published on our blog: https://blog.flude.guide/blog/libclang-ast-reconstruction-out-of-line
Also read us:
Top comments (0)