The quest for a better markdown processor (5 Part Series)
This is a quest I've been on for a long time, but the brunt of it happened in the last few days.
The first Python markdown processor I found was Trent's python-markdown2, which I got a lot of use out of, but I've since come across several issues. Backticks not working inside links and needing to edit the source to support custom elements.
There are also some minor inconveniences, like that the
markdown="1" attribute on block-level elements only works correctly if the start and end tag of the element are each on a line by themselves. Just an inconvenience for
<div class="highlight">. (It also clobbered classes from
<p> tags, which was why I was using
I looked into python-markdown a bit. It doesn't have the issue with backticks inside anchor text, but I was still unable to get the block-level custom elements working, so I didn't bother switching.
Next I found mistune. I was ready to jump on it when I found out it solved both of the two main problems, but before I did the migration, I came upon a couple of minor issues:
Although it did the Markdown-in-block-HTML processing automatically instead of requiring an attribute (which I really like), it required a blank line between the beginning and end tags and the content, as opposed to just a newline like python-markdown2. An inconvenience for my use case, but not a deal-breaker.
It didn't support
***. Double-emphasis had to be done with
**_ _**. This also isn't a deal-breaker for me, as I could write a script to convert all my existing pages and comments to the
**_syntax, but I want to have the
***syntax available if possible.
So I looked back at python-markdown a bit. It turned out it only took a single line of code (
markdowner.block_level_elements.append('expand-note')) to solve the custom element problem as far as I was concerned. With python-markdown2, this didn't work because the regex was computed as a class attribute based on the list of block elements (which meant it didn't recompute when I subclassed it or changed the list of block level elements), so I'd ended up with an ugly temporary solution of editing python-markdown2's source (obviously not acceptable long term). This made python-markdown an option for me, and it didn't have the two minor problems mistune did.
The only major obstacle left was replacing python-markdown2's ability to slugify heading text into IDs for section links. It looks to me like mistune doesn't have this, but I found out python-markdown does:
I noticed later that
python-markdown2 is also the only one of the three that supports spoilers, as far as I can tell. I could implement those in the other two with an extension, but that would require reading a lot of their code to fill the gaps in the scant documentation. And it would be very difficult to know I wasn't breaking something else.
Well, whatever. I was going to switch in python-markdown anyway. I wasn't really using Markdown spoilers, so they could wait. Then I found out that fenced code blocks don't work inside
<li>s! And I checked out the pymdownx package that folks there mentioned, but it was wrapping my code blocks in
<div class="highlight"> for who knows what reason. Of course, I could've renamed the class I was using for highlights, but I still didn't want this in my HTML output. And I couldn't find any way to disable it; even turning off the
highlight_code config didn't seem to stop it.
Frustrated, I looked at mistune some more. I found python-slugify which meant I could solve the heading ID problem. Unfortunately, mistune has some version of the above problem as well: nested fenced code blocks get extra spaces, and indented ones can work only if I remove the blank line after, which I don't want to do.
But that made me realize something: I hadn't tried indented code blocks with python-markdown. Apparently, they don't have the problem fenced ones do.
So as of now I'm finally going to switch everything to python-markdown and make the handful of edits necessary. I have low expectations that I won't come across some other issue in the first few days, but who knows. Time will tell.
Claim your page on DEV before someone else does
Level up every day