Multi-File Org-Babel Tangles with Include Directives

I've recently been using org-mode with emacs as a literate programming environment. The way it works is that you embed source blocks in your org files, similar to markdown, but with metadata that says which file to write the output to - and if you did everything right, you can output the source files by running org-babel-tangle on the file by mashing C-c C-v t.

One use case I had for this was working on LeetCode problems. I'm slowly gearing up to get a new job (hire me) and so I've been doing the odd problem here and there to stay sharp. In particular, I was working on Container With Most Water. I will try not to spoil it for people, but I'm going to be using the work I did there as my example.

So for instance, here's a snippet of ./problems/11_container_with_most_water.org from my git repo that shows what a source block might look like:

* Candidate Class
This class is used for tracking lower and upper bounds on area and for finding
the "best" candidates to test by sorting.

#+BEGIN_SRC python
@functools.total_ordering
class Candidate:
    # Pretend that there's a big ol' class definition here - No spoilers!
    pass
#+END_SRC

In this particular instance, all of the code in this file tangles to the same source output, so the metadata is defined in the org file's header at the very top of the file:

#+TITLE: Problem 11: Container With Most Water
#+PROPERTY: header-args      :tangle "../dist/11_container_with_most_water.py"

With this configuration, if I tangle the file, which lives in ./problems, it'll write to ./dist/11_container_with_most_water.py. From there, I can copy-paste the output into Leetcode and have my solution time out when trying to solve the test case with 15,000 entries.

This is pretty cool, because I can combine my solution with a bunch of notes on things I tried, things I didn't, why I did or didn't try those things, and anything else going on in my head.

Things got more complicated though once I wanted to reuse some code. For instance, I wrote a class for tracing and debugging execution and put it in ./leettrace.org:

#+TITLE: LeetTrace - pastable snippets for debug tracing
#+PROPERTY: header-args :tangle yes

This is a collection of pastable/includable snippets for debug tracing in my
Leetcode solutions.

* Python
#+BEGIN_SRC python
def pad(item, depth, justify='left'):
    # Elided


class LeetTrace:
    enabled = True
    max_depth = None
    tag_width = 8
    iter_width = 3

    def __init__(self):
        # Elided

    def log(self, message, *args, **kwargs):
        # Prints the message + metadata to the screen

    def __call__(self, *args, **kwargs):
        # A neat trick - You can call the instance like a function

    def log_if(self, pred, message, *args, **kwargs):
        # Logs if pred is truthy

    # ...

    @contextlib.contextmanager
    def context(self, label):
        # Adds the label to logging statements inside a with block

    def iter(self):
        # Log statements include the iteration number

    def sep(self):
        # A helper to separate big logging blocks


trace = LeetTrace()
#+END_SRC

I posted the full source, plus a JavaScript implementation, in my gists. Feel free to steal this!

But this is where I ran into issues. Generally, an org file can tangle to multiple source files, but tangling multiple org files into a single source file is more difficult.

I wanted to use an #+INCLUDE directive to inline this logger/tracer into my solution:

#+INCLUDE: "../leettrace.org::*Python"

If you export this org file, org will inline everything under the "Python" headline (the * Python line in the snippet from ./leettrace.org) in the output.

Unfortunately, org-babel-tangle doesn't do this step, which is a problem.

The way I solved this was by exporting my org file to another org file. Org supports exporting to all sorts of file formats, and one of them is org itself. I exported the file ./problems/11_container_with_most_water.org to ./exports/11_container_with_most_water.org, and then tangled that file to ./dist/11_container_with_most_water.py.

I put this in an Emacs lisp script at ./build.el, which looks like this:

(require 'seq)
(require 'org)
(require 'ob-tangle)
(require 'ox-org)

(let* ((problems (expand-file-name "./problems/"))
       (exports (expand-file-name "./exports/"))
       (dist (expand-file-name "./dist/"))
       (filenames
        (seq-filter
         (lambda (f) (string-suffix-p ".org" f))
         (directory-files problems)))
       (acc nil))
  (dolist (f filenames acc)
    (let ((problem-file (concat problems f))
          (export-file (concat exports f)))
      (progn
        (format-message "Exporting %s to %s..." problem-file export-file)
        (with-current-buffer (find-file-noselect problem-file)
          (org-export-to-file 'org export-file))
        (with-current-buffer (find-file-noselect export-file)
          (org-babel-tangle))))))

I can then run this file with emacs.exe --batch --load build.el. Props to u/celeritasCelery on Reddit for helping me golf out a use of s.el, ensuring that I didn't have to manually add libraries to Emacs' load path!

Comments

[Ryan McGuire]

Org supports exporting to all sorts of file formats, and one of them is org itself.

Thank you! I almost thought this was going to work for me, but I can't use the #+PROPERTY: header-args line, because I have customized header-args for each code block individually. Each of my code blocks should be tangled to a different file. The only problem is that these block-level header args appear to be stripped out (!) by (org-export-to-file 'org export-file).

This, from the include file :

#+begin_src yaml :noweb yes :eval no :tangle src/thing.yaml
thing: whatever
#+end_src

Becomes this, in the exported org document (Args are stripped!) :

#+begin_src yaml
thing: whatever
#+end_src

So therefore no code blocks are tangled because none have the :tangle argument :(

[Richard Riley]

Did you find a way to ensure that #+included files are tangled with org-babel-tangle too? It's not doing so for me. I chopped a lump out of the master doc and then stuck it in its own org file and #+include it, but org-babel-tangle doesn't tangle the #+include.