DEV Community

Cover image for Nobody told me to ship it. I shipped it anyway.
Nathan C.
Nathan C.

Posted on

Nobody told me to ship it. I shipped it anyway.

#showdev #opensource #programming #python

Six days ago I posted Your Python code is already assembly. I made it runnable.

It ended with a question. One word in the comments, I said. ship or stupid.

Comments: 0.

Two reactions. I'm fairly sure one of them was a bot and the other one was my own thumb.

So I made an executive decision.

$ pip install pyasm-lang
$ pyasm main.pya
Hello, World!
Enter fullscreen mode Exit fullscreen mode

PyAsm v0.0.1 is out. Public repo, tagged release, CI on Linux, macOS and Windows.

👉 github.com/Natuworkguy/PyAsm

Wait, what is it

Thirty second version, for anyone who missed the first post.

This is a file. It's called main.pya. It is a complete, working program:

LOAD_GLOBAL 1 (print + NULL)
LOAD_CONST 0 ('Hello, World!')
CALL 1
POP_TOP
RETURN_CONST (None)
Enter fullscreen mode Exit fullscreen mode

That is not pseudocode. That is the exact text dis.dis() prints, saved to a file, and run.

dis output has always been a receipt. Something Python hands you on the way out. PyAsm makes it an input: it parses that text and translates every instruction into ordinary Python that walks the same value stack the interpreter would, so LOAD_CONST becomes _st.append(...) and POP_TOP becomes _st.pop(). There's a --dump-python flag that shows you the whole trick, and the file it writes has no PyAsm import in it at all.

In fact, that file has no imports. None from the standard library. No dependencies from PyPI. Nothing.

That post explains the how. This one is about what happened when I actually let it out of my laptop.

Also: sorry for the silence

Six days without a post. I wasn't procrastinating, I was doing the boring part: writing a README, tagging a release, standing up a CI matrix of three operating systems and three Python versions, and designing a big new project (coming soon).

Which is where it got interesting, because shipping it found three bugs that my machine was structurally incapable of finding. All three had been sitting in the code the entire time I was writing that first post, quietly working fine on exactly one computer.

Here they are, worst first.

Bug 1: Windows ate my docstring

Every file PyAsm generates opens with a docstring saying where it came from:

"""Generated by PyAsm from main.pya.

This file is a faithful translation of the assembly,
not idiomatic Python: it walks the same value stack the
interpreter would.
"""
Enter fullscreen mode Exit fullscreen mode

Harmless. Nice, even. Then the Windows job came back and twelve tests were red at once:

main.pya: error: internal error: generated invalid Python
((unicode error) 'unicodeescape' codec can't decode bytes in
position 26-27: truncated \UXXXXXXXX escape)
Enter fullscreen mode Exit fullscreen mode

Look at where the tests put their scratch files:

C:\Users\runneradmin\AppData\Local\Temp\tmp8f2a\hello.pya
Enter fullscreen mode Exit fullscreen mode

Look at \U.

I pasted a path into a Python string literal, and \U is the start of an eight digit unicode escape. C:\Users isn't a path once it's inside quotes. It's a syntax error with a trench coat on.

The fix is three lines:

def _docstring_text(text: str) -> str:
    """Make *text* safe to paste inside a triple-quoted docstring."""
    return text.replace("\\", "\\\\").replace('"""', '\\"\\"\\"')
Enter fullscreen mode Exit fullscreen mode

Here's the part that actually rattled me. My program's entire job is generating Python source, which means every value I interpolate into it is a value being injected into a language. I'd written a careful literal-escaping path for user constants and then hand-rolled f'"""Generated by PyAsm from {name}.' for the header without a second thought, because on macOS and Linux a path is just letters and slashes.

Every backslash-free day was a day the bug looked like working code.

Bug 2: Python 3.13 popped the iterator twice

Next up, three round-trip tests exploding on 3.13 only, inside the generated module:

IndexError: pop from empty list
Enter fullscreen mode Exit fullscreen mode

Every one of the three had a for loop in it. Here's the tail of a loop, disassembled on 3.13:

   L3:    74  END_FOR
          76  POP_TOP
          78  RETURN_CONST  5  (None)
Enter fullscreen mode Exit fullscreen mode

Now the same thing across three releases of CPython:

Version How the loop cleans up its exhausted iterator
3.12 END_FOR pops it
3.13 END_FOR does nothing, the POP_TOP after it pops
3.14 END_FOR does nothing, the POP_ITER after it pops

My code asked exactly one question: is POP_ITER anywhere in this program? If yes, 3.14, so END_FOR is a no-op. If no, pop.

Which is right on 3.12, right on 3.14, and confidently wrong on 3.13, where END_FOR popped the iterator and then the POP_TOP right behind it reached into an empty stack.

The fix stops guessing the interpreter version and just looks at what's actually next to the instruction:

def _end_for_cleans_up(program: Program) -> bool:
    """Whether ``END_FOR`` has to pop the exhausted iterator itself."""
    for index, instruction in enumerate(program.instructions):
        if instruction.opname != "END_FOR":
            continue
        after = program.instructions[index + 1:index + 2]
        if after and after[0].opname in ("POP_TOP", "POP_ITER"):
            return False
    return True
Enter fullscreen mode Exit fullscreen mode

The lesson I'm taking: when CPython changes bytecode, it doesn't only rename opcodes. It moves stack effects between them. An opcode table that only knows names will pass every test you write on the version you happen to be running.

Bug 3: 3.11 doesn't have inlined comprehensions

Last one, and this one isn't a bug in my code so much as physics.

On 3.11, this round-trip test died:

error: cannot assemble the constant
<code object <listcomp> at 0x106a16bf0, file "<snippet>", line 2>
hint: constants must be literals; code objects, and the functions
and classes built from them, are out of scope
Enter fullscreen mode Exit fullscreen mode

The snippet was one line:

squares = [n * n for n in range(6) if n % 2]
Enter fullscreen mode Exit fullscreen mode

PEP 709 inlined comprehensions in 3.12. Before that, a list comprehension compiled to its own separate code object, and a separate code object shows up in dis output as <code object <listcomp> at 0x7f...>.

You cannot rebuild an object from its repr. The address isn't the object, it's a rumour about where an object used to be. So on 3.11 that comprehension is genuinely unassemblable, and the honest fix is a skip that says so out loud:

if name in INLINED_FROM_312 and sys.version_info < (3, 12):
    self.skipTest(
        "before 3.12 a comprehension compiles to a nested "
        "code object, which is out of scope (PEP 709)"
    )
Enter fullscreen mode Exit fullscreen mode

Same comprehension, on 3.12 and up, round-trips perfectly. A language feature quietly changed shape underneath me and my test suite was the thing that noticed.

What actually shipped

  • 152 opcodes, covering 3.11 through 3.14 spellings, both calling conventions, LOAD_SMALL_INT, TO_BOOL, POP_ITER, the lot
  • Zero dependencies. Nothing. Not one.
  • 79 tests, including full Python -> assembly -> Python round trips that assert the output matches plain Python character for character
  • CI on Linux, macOS and Windows, three Python versions each, which is the only reason this post has three war stories in it
  • A real CLI: run, dump, check, dis, opcodes
  • Errors that point at your assembly, with a caret, even when the failure happens deep inside a runtime helper

One naming quirk to save you a confused minute: it installs as pyasm-lang and imports as pyasm. The short name was already taken on PyPI by an unrelated project from years ago, so the distribution got the suffix and the package kept the name you'd expect.

>>> from pyasm import run
>>> _ = run("LOAD_NAME (print)\nPUSH_NULL\nLOAD_CONST ('hi')\nCALL 1")
hi
Enter fullscreen mode Exit fullscreen mode

Yes, there's a Python API too. assemble, run, disassemble_source, and the .pya text goes in as a plain string.

Try it in 60 seconds 

$ pip install pyasm-lang
Enter fullscreen mode Exit fullscreen mode

Write a program by hand, with labels instead of byte offsets:

        LOAD_CONST 3
        STORE_NAME n

loop:   LOAD_NAME n
        TO_BOOL
        POP_JUMP_IF_FALSE done
        LOAD_NAME print
        PUSH_NULL
        LOAD_NAME n
        CALL 1
        POP_TOP
        LOAD_NAME n
        LOAD_CONST 1
        BINARY_OP (-)
        STORE_NAME n
        JUMP_BACKWARD loop

done:   LOAD_NAME print
        PUSH_NULL
        LOAD_CONST ('Liftoff!')
        CALL 1
        POP_TOP
        RETURN_CONST (None)
Enter fullscreen mode Exit fullscreen mode

Or take the shortcut and let Python write it for you:

$ pyasm dis fizzbuzz.py -o fizzbuzz.pya
$ pyasm fizzbuzz.pya
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
...
Enter fullscreen mode Exit fullscreen mode

And when you want to see how the sausage is made:

$ pyasm dump fizzbuzz.pya -o generated.py
$ python generated.py    # no pyasm import anywhere in it
Enter fullscreen mode Exit fullscreen mode

Still can't do it

Unchanged from last time, and for the same reason both times: text disassembly doesn't carry everything a code object has.

try/except is driven by a code object's exception table, and dis doesn't print one you can reconstruct. raise works fine.

def, class, lambda compile to separate code objects, which is bug 3 above wearing a different hat. You can import and call any Python function you like, you just can't define one inside a .pya file.

Both of these are on the table if enough people want them. Both are real work.

The ask, redesigned

Last time I asked for one word and got zero words, which is a completely fair review of my ask. So here's a better one.

Write me the most cursed valid .pya file you can, and put it in the comments.

That's it. Anything that assembles. Hand-written assembly that shells out. A quine. FizzBuzz with no %. Something that abuses the NULL slot in a way I didn't anticipate and files a bug for me. I will run every single one of them, and I'll reply to every single one.

Star it if you want to see where it goes: github.com/Natuworkguy/PyAsm

Issues are open too, and honestly, "this crashed" is my favourite kind of issue right now. Three of them just made the thing meaningfully better and they all came from a computer I don't own.

What's the first thing you'd break? 👇

Top comments (0)