DEV Community: kAZUYA tAKEI

Render "GFM"-likely HTML in Sphinx with MyST-Parser

kAZUYA tAKEI — Sun, 11 May 2025 18:37:07 +0000

MyST-Parser is Sphinx extension to parse Markdown document sources.
We can write high structured document using it even if we don't know syntax reStructuredText.

MyST-Parser supports GFM, but ...

MyST-Parser has some options for behavior. ¹
There is myst_gfm_only in options. This is to run parse GitHub Flavored Markdown (GFM) that support some extended syntax from Commonmark.

But, when this option is True, it does not inject <br> into linefeed of sources.

Example

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

This markdown source has line feed each sentence.

When Sphinx converts it to HTML by sphinx-build.

<p>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>

But GitHub render from this source. ²

<p>
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
<br />
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
<br />
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
<br />
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>

Why it happened?

GFM is not matched between spec and behavior.

GFM spec is put on https://github.github.com/gfm/.
There are "6.12Hard line breaks
" and "6.13 Soft line breaks" about spec of line breaks.

Spec of soft line break:

A regular line break (not in a code span or HTML tag) that is not preceded by two or more spaces or a backslash is parsed as a softbreak. (A softbreak may be rendered in HTML either as a line ending or as a space. The result will be the same in browsers. In the examples here, a line ending will be used.)

We expect that it renders nothing when there are soft line breaks.
But actually, GitHub.com renders <br> tag from line feed.

MyST-Parser doesn't have options for line breaks.

It is difficult that to adjust for implementation of GitHub?
No, you can resolve it by hack class definition.

How to hack for realize "actual" GFM style line breaks.

Code

Please write it code into conf.py of your document.

def setup(app):
    # If you already defined setup, write these code into before `return`. 
    from myst_parser.mdit_to_docutils.base import DocutilsRenderer

    DocutilsRenderer.render_softbreak = DocutilsRenderer.render_hardbreak

Description

When Sphinx with MyST-Parser convert from Markdown to HTML, generates intermediate object called "doctree".

MyST-Parser parse Markdown and create doctree.
Writer process of Sphinx generates HTML from doctree.

To create doctree from Markdown, MyST-Parser defines DoctreeRenderer.
This class have methods that works each AST nodes of Markdown.

DocutilsRenderer.render_softbreak is method to put linefeed code when process visits "soft line breaks".
DocutilsRenderer.render_hardbreak is method to put HTML when process visits "hard line breaks".

In Python, we can override methods of classes by reassign.
Resolve code is to override render_softbreak by render_hardbreak.

Therefore, when parser found all line breaks, it runs as "hard line breaks".

https://myst-parser.readthedocs.io/en/latest/configuration.html ↩
Actually, <br> tag puts on single line, it added tail of lines. ↩

Hosting presentation on Read the Docs

kAZUYA tAKEI — Sat, 19 Oct 2024 06:36:25 +0000

This is updates of previous post.

About Read the Docs

Read the Docs(RTD) is "documentation platform" to host website about document of software.
RTD can integrate with GitHub. If integrated repository has .readthedocs.yaml, build sources and deploy contents when it is pushed.

Example of `.readthedocs.yaml` for standard Sphinx:

version: 2

build:
  os: 'ubuntu-22.04'
  tools:
    python: '3.11'

sphinx:
  configuration: docs/conf.py
  builder: dirhtml

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

On push event, RTD runner works these:

Set up Python 3.11 (ref: build.tools)
Install dependencies by docs/requirements.txt (ref: python.install)
Build document by dirhtml builder using docs/conf.py (ref: sphinx)

Custom build supportings

There is build.commands property in .readthedocs.yaml that define procedures to build contents from source.
as far as it outputs contents into specified path, RTD deploy it as published site.

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.12"
  commands:
    - pip install pelican
    - pelican --settings docs/pelicanconf.py --output $READTHEDOCS_OUTPUT/html/ docs/

This is example on reference about build.commands.
It runs pelican (other website build of Python) and output $READTHEDOCS_OUTPUT/html.

Anything is ok that programs generate html files.
Even if it generate presentation.

Deploy presentation on Read the Docs

version: 2

build:
  os: 'ubuntu-22.04'
  tools:
    python: '3.11'
  commands:
    - pip install -r requirements.txt
    - sphinx-build -b revealjs source $READTHEDOCS_OUTPUT/html

This is .readthedocs.yaml of example for deploy Reveal.js presentation by sphinx-revealjs.

There is generated presentation.

Pros and Cons

Pros

Keep older presentation for users (e.g. seeing archive movie of YouTube).
- Presentation should be fixed for typo and more.
- If author update with bigger changes, user can see old version.
It can toggle version by bundle widget of RTD.
It can be visible or hidden manually.

Cons

RTD adds AD on presentation.
Presentation does not root of FQDN.

A short hack to deploy reveal.js presentation on Read the Docs

kAZUYA tAKEI — Sun, 03 Apr 2022 09:11:51 +0000

This is an introduction to serve reveal.js slide to Read the docs.

Abstract

Edit your conf.py to override target builder.

def override_builder(app, config):
    # Change for your RTD settings. Please set it in your RTD prject
    app.registry.builders["singlehtml"] = app.registry.builders["revealjs"]

def setup(app):
    app.connect("config-inited", override_builder)

You can see contents and sources.

Approaching

Problems for regular usage to server sphinx-revealjs

Read the docs are documentation platform supporting Sphinx.
You can serve HTML files from Sphinx documentation as same as run make html on local.

sphinx-revealjs is Sphinx extension to provide builder that generate Reveal.js style HTML. (ex: call make revealjs)

RTD supports only html, dirhtml and singlehtml for HTML buildings.
So, it is difficult to server reveal.js presentation of sphinx-revealjs.

Sphinx can override builder object

Sphinx provides Application API to inject procedures on some events.
config-inited event is called on after loading all configuration before creating target-builder object.

We can modify builder-class in the handler for this event, and can use any builder in RTD.
(please see abstract for example)

Caution

Cannot serve regular name builds

This hack switches the builder, and you cannot use regular builder.
You should be careful what builder you want to call.

RTD widget is not working

RTD provides widget for versioning and i18n.
But reveal.js content does not render wighet.
You can serve single version presentation only!

This hack is NOT clean

This is dirty hack.
RTD may not accept this hack in backend.

Create Errbot backend/storage plugin as PyPI package

kAZUYA tAKEI — Wed, 15 May 2019 15:04:09 +0000

In develoving some Errbot plugins, I am thinking that storage and backend plugins have better managed as package (if possible, should be register at PyPI)

1. In bootstrapping, Errbot does not install dependencies of backend and backend.

To run bot, I call errbot command. In this command, load plugins in order of storage, backend and bot-plugins.
When bootstrap load storage and backend plugin, it does not check dependencies.

Actually some plugins let to install dependencies by command-line.

ex:

If plugins have dependencies and need to install manually these, I think better to define as module and package.

2. In running, Errbot does not change dynamically storage and backend.

Errbot is running as daemon generally. for that reason, storage and backend plugin will not almost be changed in running.

Therefore, these can be managed as requirements together with Errbot core.
If this, pip install -r requirements.txt in your local repository can install all dependencies.

ex: (these plugins are not registered in PyPI)

errbot
errbot-backend-gitter
errbot-storage-firebase

Create as project

List of files

Project structure is same to standard python package.

setup.py
setup.cfg
README.rst
MANIFEST.in
errbot_backend_demo
- __init__.py
- demo.py
- demo.plug

Writings

`setup.py`

It can write simply.

#!/usr/bin/env python
from setuptools import setup

setup()

`setup.cfg`

Currently, all metadata can write in setup.cfg.
Please define options.package_data section and include_package_data key to include .plug file as installation resource.

[metadata]
name = errbot-backend-demo
version = attr:errbot_backend_demo.__version__
author = Kazuya Takei
author_email = attakei@example.com
description = Demo backend plugin for Errbot
long_description = file:README.rst

[options]
install_requires = YOUR_DEPENDENCIES
packages = find:
include_package_data = True

[options.package_data]
errbot_backend_demo = *.plug

`LICENSE.rst`

Write normally.

`MANIFEST.in`

As same as setup.cfg, define .plug as include file.

include errbot_backend_demo/*.plug

`init.py`

Define version of package here. And define function to return path of installation path.

from pathlib import Path


__version__ = '0.0.1-alpha.1'


def get_plugin_dir() -> str:
    module_dir = Path(__file__).parent
    return str(module_dir)

`demo.py` and `demo.plug`

Write normally as plugin.

Use in Errbot config.py

If plugin define as package by these, you can write config.py to use plugin.

import errbot_backend_demo

BOT_EXTRA_BACKEND_DIR = errbot_backend_demo.get_plugin_dir()
BACKEND = 'Demo'

Sphinx extension to generate Reveal.js presentation from plain reST

kAZUYA tAKEI — Mon, 12 Nov 2018 07:41:24 +0000

I create Sphinx extension to convert Reveal.js presentation from plain reStructuredText. please try it, if you match want to write Reveal.js by Sphinx(reStructuredText)

attakei / sphinx-revealjs

Sphinx builder to revealjs presentations

sphinx-revealjs

Sphinx extention with theme to generate Reveal.js presentation

Orverview

This extension generate Reveal.js presentation from standard reStructuredText.

It include theses features.

Custom builder to translate from reST to reveal.js style HTML
Template to be enable to render presentation local imdependent

Installation

$ pip install sphinx-revealjs

Usage

Create your sphinx documentation
Edit conf.py to use this extension
```
extensions = [
    'sphinx_revealjs',
]
```
Write source for standard document style
Build sources as Reveal.js presentation
```
$ make revealjs
```

Change logs

See it

Futures

Index template as none presentation
CDN support

Copyright

Apache-2.0 license. Please see LICENSE.

View on GitHub

If you want to see source and output, click these links. These are pair of simple reST source and generated presentation.

Source from
- https://raw.githubusercontent.com/attakei/sphinx-revealjs/master/demo/index.rst
Output to
- https://attakei.github.io/sphinx-revealjs/index.html

Installation and usage

Installation

$ pip install git+https://github.com/attakei/sphinx-revealjs

Usage

Add extension to your sphinx conf.py

extensions = [
    'sphinx_revealjs',
]

Write your presentation source by reST, and build

$ make revealjs

Concepts

Use default nodes and directives as more as possible
Manage multiple presentations in one Sphinx documentation
- and, can switch themes per documents

Supporting features in reST

Section (convert reveal.js section)
Comment (use for speaker note)
Code block (convert reveal.js style code block)
And other directives (inherit Sphinx HTML5 builder)

Short examples for convert

Your write source like this..

=====
Title
=====

Section 1
=========

content 1

Section 2
=========

content 2

It can convert this

<section>
  <h1>Title</h1>
</section>
<section>
  <h2>Section 1</h2>
  <div>content 1</div>
</section>
<section>
  <h2>Section 2</h2>
  <div>content 2</div>
</section>

Of course, support nested section.

from

=====
Title
=====

Section 1
=========

content 1.1
-----------

content body

content 1.2
-----------

content body

<section>
  <h1>Title</h1>
</section>
<section>
  <section>
    <h2>Section 1</h2>
  </section>
  <section>
    <h3>content 1.1</h3>
    <div>content body</div>
  </section>
  <section>
    <h3>content 1.2</h3>
    <div>content body</div>
  </section>
</section>

DEV Community: kAZUYA tAKEI

Render "GFM"-likely HTML in Sphinx with MyST-Parser

MyST-Parser supports GFM, but ...

Example

GFM is not matched between spec and behavior.

How to hack for realize "actual" GFM style line breaks.

Code

Description

Hosting presentation on Read the Docs

About Read the Docs

Example of .readthedocs.yaml for standard Sphinx:

Custom build supportings

Deploy presentation on Read the Docs

Pros and Cons

Pros

Cons

A short hack to deploy reveal.js presentation on Read the Docs

Abstract

Approaching

Problems for regular usage to server sphinx-revealjs

Sphinx can override builder object

Caution

Cannot serve regular name builds

RTD widget is not working

This hack is NOT clean

Create Errbot backend/storage plugin as PyPI package

1. In bootstrapping, Errbot does not install dependencies of backend and backend.

2. In running, Errbot does not change dynamically storage and backend.

Create as project

List of files

Writings

setup.py

setup.cfg

LICENSE.rst

MANIFEST.in

__init__.py

demo.py and demo.plug

Use in Errbot config.py

Sphinx extension to generate Reveal.js presentation from plain reST

attakei / sphinx-revealjs

Sphinx builder to revealjs presentations

sphinx-revealjs

Orverview

Installation

Usage

Change logs

Futures

Copyright

Installation and usage

Installation

Usage

Concepts

Supporting features in reST

Short examples for convert

Example of `.readthedocs.yaml` for standard Sphinx:

`setup.py`

`setup.cfg`

`LICENSE.rst`

`MANIFEST.in`

`init.py`

`demo.py` and `demo.plug`