DEV Community: Jones Beach

Write your first library

Jones Beach — Mon, 17 Nov 2025 12:30:00 +0000

At my first real software internship, the highlight of the summer was a 24-hour hackathon. My team and I sat in a conference room for way too many hours, hacking together a prototype. I barely remember the end goal. A chat something?

What I recall most was the feeling of trying one JavaScript library after another, failing to get basic tabs to work. I felt helpless!

Clearly I needed more JS experience and should’ve asked for help, but there’s a different point I want to emphasize: the shift from being a library user to being a library writer.

What is a library?

You’ll see different terms depending on the person, language, and ecosystem: library, package, crate, SDK. These all represent code which is already written in your language which you can pull in freely to your project. I prefer the term library because it’s timeless, and would make Andrew Carnegie proud (I hope!).

The size can vary anywhere from a small date/time utility to a massive framework like React. Each of these is a library and was written by someone with a keyboard like you and me.

The code itself typically lives on GitHub, but could be anywhere, and a built/packaged version of that code lives in an online package index. For JavaScript, this is NPM, in Python it’s PyPI, in Rust it’s crates.io.

It’d be easy to get lost in the licensing details or best practices, but others have covered this better and more deeply than I will.

I’m going to make the case for why you should write your own, and how it will transform your relationship with your code.

You’ll feel more confident in your stack.

The vague uncertainty and powerlessness I felt during the hackathon? I had no idea what I was doing.

Each JavaScript library I found on the internet, each example I stumbled through, all I could think was “I hope this one works.” I was at the whim of what was published, potentially incomplete, years prior.

After you’ve written even one library, your thinking shifts to: “I understand the general shape of what’s happening here.”

When you see an import, you’ll know roughly what that maps back to on the file system, and how they exposed that set of symbols (functions, classes, etc) publicly.

You don’t need to be able to regurgitate the pseudocode behind the scenes. Just knowing you can navigate their repo turns what used to be a black box into just another piece of code in your system, this time written by someone else.

Once the black box feeling dissolves, something interesting happens: you start forming opinions. You start noticing which libraries feel intuitive and which feel clunky. That’s discernment, and it naturally leads you to the next question: what makes a good interface in the first place?

Your discernment around other libraries will increase.

The conventional wisdom for picking a library is to check their GitHub stars (a popularity contest for nerds), version number (be wary of anything below 1.0!), and how recent its commits are. This is all valid, but I’d encourage you to go one step further: look at their examples and see if they match your mental model of the problem you are trying to solve. This forces you to think through what you need the library for in the first place. Sometimes you'll think “uhhhh this looks more complicated than I expected,” other times "ah perfect, they make this super easy."

A library is ultimately a contract: when you ask for X, it will do Y or give Z. A good one will do this with minimal friction.

When I pull up the docs for a new library, my first thought is about their interface. What functions and classes they expose, how they fit together. And to be honest, I often look for a bit of awe, a sense of “how did they do this?!” (I wrote about this in Rust’s Axum.)

I’m not saying every library you use needs to be fancy, elegant, or clever. But there’s a sense of “does this match what I was expecting it to do?” that’s always worth asking yourself.

The moment you can recognize a clean interface in someone else’s library, you’re ready to design your own.

You’ll learn to design interfaces.

You may recall when you wrote your first function and called it. Code reuse! Writing a library is exactly the same, just with harder interfaces and more pomp and circumstance.

Each time you think “this function could really use another parameter,” that’s interface work. Each time you realize a function printed an error rather than throwing or returning an error to the caller, that’s interface work.

Designing a usable interface is as much art as science. A good one will hide most of the details and expose a small surface area of knobs for the user at the right time. This will feel difficult and awkward at first, but with time you’ll get the hang of it.

Once you’ve designed your first library interface, you’ll start to notice places within your projects where a harder interface is more useful. The voice I often hear is “hmm this file shouldn’t need to know about this,” and it’s a sign my interface could be stronger.

At this point, the difference between “code I wrote and am now reusing” and “code someone else wrote and I’m now reusing” starts to go away.

It’s all just code, available for your next idea.

It’s empowering.

This is the real reason. Everything else about saving time and not reinventing the wheel is valid and probably what you should tell your manager.

There’s a deeper truth here. Early on in your career, you will use libraries to build applications. You may even have users, and hopefully your code is solving a problem for them.

When you write a library, your users are other engineers like yourself. If your goal is to build an app that changes the world, this isn’t the fastest path. But if you’re driven to feel like a real engineer, someone fully in control of their tools and their craft, this is the way.

Some people may say writing a library will take 10 pounds off and make people laugh harder at your jokes. I can’t speak to that, but it will give you something equally elusive: agency. The agency I was severely missing sitting in the conference room praying one of the libraries I found would finally work. No more fingers crossed that the next one will magically work. You get to be the person who writes the thing others depend on.

This post is about libraries, but this is also why I started From Scratch Code. This skill set we share, which is sometimes valued by the job market and sometimes not, is something no one can take away from you. Why not use it to add to the canon of code in the world?

Writing a library is the best first step to own your craft deeply. If you want to get started and aren’t sure how, I’d love to help.

Subscribe & Save [on nothing]

Want a software career that actually feels meaningful? I wrote a free 5-day email course on honing your craft, aligning your work with your values, and building for yourself. Or just not hating your job! Get it here.

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

I also write essays and fiction about neurodivergence, meaningful work, and building a life that fits. My novella Lake-Effect Coffee is a workplace satire about burnout, friendship, and a coffee van. Read the first chapter or grab the ebook.

Memphis goes online: adding socket support

Jones Beach — Mon, 03 Nov 2025 12:30:00 +0000

Before we begin, let’s agree that “online” in this context means accepting bytes from localhost. Deal? Deal.

You may recall that sometime last century I had the brilliant idea to boot a Flask server from inside Memphis. Trudging toward that goal exposed me to a ton of types and functionality I didn’t yet support. I even ran an interpreter inside an interpreter! But it was a slog. Literally line 1 imports something from the stdlib which imports something from the stdlib which imports something else from the stdlib. I never got to line 2. At a certain point, I realized there was no light at the end of the tunnel.

Skip to a few weeks ago, it hit me I could make my whole life easier by writing my own HTTP library in Python, wire up the necessary system resources from the Rust side, and start running some curl commands.

This post documents Memphis’ journey from being air-gapped to responding in kind:

> curl localhost:8080
Hello from Enid (powered by Memphis)

We’ll approach this like this:

What we want a net module for Memphis to look like
How I implemented this
Introducing Enid, a micro HTTP framework to wrap this up (and allow us to pretend we got Flask working all along)

A `net` module for Memphis

The surface area to respond to an HTTP request is surprisingly small. (Maybe that’s why I keep reimplementing them? Let’s put a pin in that for later.)

Essentially, we need to:

Listen on a port
Accept new connections on that port

That’s it!

Here’s a minimal interface to do this.

from memphis.net import listen

sock = listen(("127.0.0.1", 8080))
print("Listening...")
conn, _ = sock.accept()
data = conn.recv(1024)
print("Sending...")
conn.send(b"HTTP/1.1 200 OK\r\n\r\nHello from Memphis!")
print("Closing...")
conn.close()

If you want to skip ahead, here’s a short video showing this in action.

This code should block on the sock.accept(), just like any socket-based server, until it receives a connection on localhost:8080.

In other words, this is the simplest possible Memphis web server, one that says hello over TCP. And we format the bytes we send back as HTTP so we can test this with curl.

You’ll notice we’re importing memphis.net.listen rather than socket like you would in CPython. That’s what we have to build. From scratch!

To do so, we’ll need:

a memphis.net module, which contains
a listen function, which, when called, returns
a Socket object, which contains an accept method which, when called, returns
a Connection object, which contains recv, send, and close methods.

That’s a lot to wire up! But we can split this into two parts:

the wiring up of that laundry list
connecting the Socket and Connection objects to use Rust utilities to actually do things

This is the paradox of implementing a programming language: you spend 90% of your time on the piping, and then just call your host language to do the real work.

Here our tools are Rust’s TcpListener and TcpStream. Let’s take a look at how we can connect those to our Python code.

Implementing the `net` module

1. Building the module

Our first step is to create a memphis.net module, which will contain the symbols listen, Socket, and Connection.


fn builtins() -> Vec<Box<dyn CloneableCallable>> {
    vec![Box::new(NetListenBuiltin)]
}

fn init(type_registry: &TypeRegistry) -> Module {
    let mut net_mod = Module::new(Source::default());
    for builtin in builtins() {
        net_mod.insert(&builtin.name(), TreewalkValue::BuiltinFunction(builtin));
    }

    register_native_class::<Socket>(&mut net_mod, "Socket", type_registry);
    register_native_class::<Connection>(&mut net_mod, "Connection", type_registry);

    net_mod
}

pub fn import(module_store: &mut ModuleStore, type_registry: &TypeRegistry) {
    let net_mod = init(type_registry);

    let memphis_mod = module_store.get_or_create_module(&ImportPath::from("memphis"));
    memphis_mod.borrow_mut().insert(
        "net",
        TreewalkValue::Module(Container::new(net_mod.clone())),
    );

    module_store.store_module(&ImportPath::from("memphis.net"), Container::new(net_mod));
}

We store our new module in the ModuleStore, which will allow us to find our module when a user’s code imports it.

Next, let’s take a look at the two native structs that back the classes we just registered.

2. Defining our native structs

First, here is our native Socket, which is a thin Rust wrapper.

use std::{
    io,
    net::{SocketAddr, TcpListener, TcpStream},
};

pub struct Socket {
    listener: TcpListener,
}

impl Socket {
    pub fn new(host: String, port: usize) -> io::Result<Self> {
        Ok(Self {
            listener: TcpListener::bind(format!("{}:{}", host, port))?,
        })
    }

    pub fn accept(&self) -> io::Result<(TcpStream, SocketAddr)> {
        self.listener.accept()
    }
}

And our native Connection, another thin Rust wrapper.

use std::{
    io::{self, Read, Write},
    net::{Shutdown, TcpStream},
};

pub struct Connection {
    stream: TcpStream,
}

impl Connection {
    pub fn new(stream: TcpStream) -> Self {
        Self { stream }
    }

    pub fn recv(&mut self, bufsize: usize) -> io::Result<Vec<u8>> {
        let mut buffer = vec![0; bufsize];
        let n = self.stream.read(&mut buffer)?;
        Ok(buffer[..n].to_vec())
    }

    pub fn send(&mut self, data: &[u8]) -> io::Result<()> {
        self.stream.write_all(data)
    }

    pub fn close(&mut self) -> io::Result<()> {
        self.stream.shutdown(Shutdown::Both)
    }
}

It’s worth mentioning that these are both synchronous. I could integrate these with tokio down the road, but that felt like a can of worms best left for future me.

These structs encapsulate the socket behaviors we need, but we still need to connect those up to their Python bindings. You’ll notice these two structs don’t reference our actual interpreter logic: no Module, no knowledge they will be used to evaluate Python code. That’s on purpose!

To make them useful, we’ll bridge our native structs to our Python type system.

3. Wiring them into the type system

In the snippet below, we register our builtin methods to Socket and Connection. impl_method_provider is a macro to reduce the boilerplate to assign our methods to a type.

impl_method_provider!(Socket, [AcceptBuiltin]);
impl_method_provider!(Connection, [ConnRecv, ConnSend, ConnClose,]);

fn register_native_class<T: MethodProvider>(
    mod_: &mut Module,
    name: &str,
    type_registry: &TypeRegistry,
) {
    let object_class = type_registry.get_type_class(&Type::Object);
    let type_class = type_registry.get_type_class(&Type::Type);

    let mut class = Class::new_direct(name, Some(type_class.clone()), vec![object_class.clone()]);

    for builtin in T::get_methods() {
        class.set_on_class(&builtin.name(), TreewalkValue::BuiltinMethod(builtin));
    }

    mod_.insert(name, TreewalkValue::Class(Container::new(class)));
}

In register_native_class, we create a Python Class for each and add these two new classes to our new memphis.net module, each with the specified builtin methods.

With this in place, we can finally begin filling out our builtin methods, starting with NetListenBuiltin.

4. Adding builtin functions

In Memphis, all builtin functions or methods are just structs which implement the Callable trait. This trait requires the struct provide its name and implement a call method. The struct itself doesn’t need any fields because those will all be passed in as runtime parameters.

When a builtin is invoked, it’s provided a reference to the interpreter &TreewalkInterpreter and any arguments Args. The former is needed to access any other runtime resources or raise errors, the latter because that’s how functions work.

pub struct NetListenBuiltin;

impl Callable for NetListenBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        check_args(&args, |len| len == 1, interpreter)?;

        let host_port = args.get_arg(0).as_tuple().raise(interpreter)?;
        let host = host_port.first().as_str().raise(interpreter)?;
        let port = host_port.second().as_int().raise(interpreter)?;

        let socket = Socket::new(host, port as usize)
            .map_err(|e| interpreter.runtime_error_with(format!("Failed to bind Socket: {}", e)))?;

        let socket_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Socket"))
            .ok_or_else(|| interpreter.runtime_error_with("Socket class not found"))?;

        let obj = Object::with_payload(socket_class.clone(), socket);

        Ok(TreewalkValue::Object(Container::new(obj)))
    }

    fn name(&self) -> String {
        "listen".into()
    }
}

In this builtin, we:

evaluate and type check the positional arguments
create a new Socket
look up the memphis.net.Socket class
create an Object of this class and provide it a Socket

For this last step to work, we must add native object support to our Python Object. Let’s take a look at that now.

5. Storing native payloads on objects

We start by adding a new field to our Object, an optional Box<dyn Any>. I considered using an enum here, something like Option<NativeResource>, but I was stubborn and went with dynamic dispatch. I’ll explain why shortly.

pub struct Object {
    class: Container<Class>,
    scope: Scope,
    native_payload: Option<Box<dyn Any>>, // this is new!!!!!
}

We are now up to the sock.accept() call.

struct AcceptBuiltin;

impl Callable for AcceptBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        let self_val = args.get_self().raise(interpreter)?;
        let socket = self_val.as_native_object::<Socket>().raise(interpreter)?;

        let (stream, addr) = socket.accept().map_err(|e| {
            interpreter.runtime_error_with(format!("Socket.accept() failed: {}", e))
        })?;

        let conn = Connection::new(stream);

        let conn_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Connection"))
            .ok_or_else(|| interpreter.runtime_error_with("Connection class not found"))?;

        let conn_obj = Object::with_payload(conn_class.clone(), conn);

        Ok(TreewalkValue::Tuple(Tuple::new(vec![
            TreewalkValue::Object(Container::new(conn_obj)),
            TreewalkValue::Str(Str::new(&addr.to_string())),
        ])))
    }

    fn name(&self) -> String {
        "accept".into()
    }
}

The key line here is as_native_object::<Socket>(), which reads our new Object.native_payload field and attempts to downcast it to a Socket object.

If I had gone with an enum, I could have used an as_socket() pattern or similar, but I just didn’t want to. There are other potential benefits to using Box<dyn Any>, like dynamic registration of native types, but that feels a ways off.

I won’t show the recv, send, and close methods on Connection, but they work very similarly to accept.

With that, our memphis.net module is wired up and implemented. Thanks for sticking with me! I know those code snippets were a slog.

To recap, we:

created a new memphis.net module
populated the module with a listen builtin function and Socket and Connection classes
added native object support to our Object, with the ability to downcast to retrieve the native objects
created a native Socket and Connection using TcpListener and TcpStream in Rust
created builtins for Socket.accept, Connection.recv, Connection.send, and Connection.close

Before we stop, let’s make our HTTP server a bit more fun.

Introducing Enid

With the net module hooked up and working, I wanted to make my HTTP calls look more like Flask, with its decorator usage and general clean feel. I chose to call this Enid, because that’s another small town in the middle of the US.

Our new API looks like this.

from enid import App, Response

app = App()

@app.get("/")
def home(req):
    return Response.text("Hello from Enid (powered by Memphis)")

if __name__ == "__main__":
    app.run()

Any calls to the root / should return a 200 with a text response, everything else should return a 404. Here’s an example curl session.

> curl localhost:8080
Hello from Enid (powered by Memphis)
> curl localhost:8080/
Hello from Enid (powered by Memphis)
> curl localhost:8080/another_endpoint
404 Not Found
> curl localhost:8080/anything_else
404 Not Found
> curl localhost:8080
Hello from Enid (powered by Memphis)

Wrapping up

Implementing socket support also revealed several Python details I’d missed. I didn’t support __name__, I had a ROUGH kwargs bug, plus I needed to implement str.encode, str.join, str.split, and bytes.decode.

If you want to try it on GitHub, you can run an Enid app on either Python or Memphis. With ChatGPT’s assistance, I wrote a shim so that it will try to import from Memphis, then fallback to Python’s stdlib, like so:

try:
    from memphis.net import listen
except ImportError:
    # Fallback to the pure-Python shim
    from .shim_net import listen

This whole project took about two weeks, which is massively shorter than if I’d kept on the road to Flask nirvana. The wiring to reach the inside of the listen function took about a day, getting the rest of the stdlib and pipes working about a week, and the last week was integrating it into the type system.

To show why the type system work was necessary, this snippet shows two things I wasn’t able to do even when I could technically respond to a curl command.

from memphis import net

type(net.Connection) # <class 'type'>
"send" in dir(net.Connection) # True

This reflection was only possible once we registered the Connection class into memphis.net and assigned it methods. This, plus adding the native payload to a Python object, was just as rewarding as seeing the curl work because it meant Memphis is growing up. And while I can’t say this was easy, I also don’t feel like the code is all going to fall apart tomorrow. So we proceed!

There’s a ton here I could do next. Maybe I’ll support route pattern matching (/endpoint/{id}) and middleware in Enid, or get this all working on my bytecode VM. I could even try to wrap my head around what an async version of this would look like. The order I tackle these will probably depend on how I feel when I wake up tomorrow.

With that, Memphis is officially online! Just don’t count on it to respond to your Slack messages.

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

Python’s operator chaining is mildly interesting

Jones Beach — Mon, 20 Oct 2025 11:30:00 +0000

Nary a week ago, I typed a harmless expression into my Memphis REPL: 4 == 4 == 4.

My REPL’s response? False.

I’m glad it can speak its mind, but the gall!

Apparently, you can’t just treat a chain of comparison operators like a tree of binary expressions. According to my BlueSky timestamps, it took me 13 minutes to figure this out. And I’m here today so you don’t make the same misunderstanding I did.

Fixing the parser

Expressions in Python are parsed in a left-associative way, meaning something like 2 + 3 + 4 will be parsed as (2 + 3) + 4. This works fine for numerical operations like addition, and we can rely on operator precedence and parentheses to control evaluation order.

If you apply this same approach to comparison operators, like I was doing, you get this: (4 == 4) == 4. Which reduces to True == 4. Which is False. Where I thought my REPL was giving me sass, it was just doing math.

How can we fix this? We need to move from my tree-based approach to a flat structure, something that records all the operators and operands in order. I ended up introducing a new Expr variant to represent these chains.

enum Expr {
    ...
    BinaryOperation {
        left: Box<Expr>,
        op: BinOp,
        right: Box<Expr>,
    },
    ComparisonChain {
        left: Box<Expr>,
        ops: Vec<(CompareOp, Expr)>,
    },
    ....
}

You may be wondering why we need to store the CompareOp in between each right operand. My first thought was that we’d only need to store it once! As I looked more into the Python rules (AKA asked ChatGPT for test cases), I learned that Python allows you to write heterogeneous chains. Take these examples.

a = 1 < 2 == 2 < 3                    # True
b = 1 == 2 != 3                       # False
c = 2 in [1,2,3] in [[1,2,3],[4,5,6]] # True

Honestly, I’d be pretty annoyed if I saw any of these in code. But Python allows them, so we must fall in line.

Here are my new parser tests, along with a Rust macro to make it easier to build these expressions.

fn operator_chaining() {
    let input = "a == b == c";
    let expected_ast = cmp_chain!(var!("a"), [(Equals, var!("b")), (Equals, var!("c")),]);

    assert_ast_eq!(input, expected_ast, Expr);

    let input = "a == b < c > d";
    let expected_ast = cmp_chain!(
        var!("a"),
        [
            (Equals, var!("b")),
            (LessThan, var!("c")),
            (GreaterThan, var!("d")),
        ]
    );

    assert_ast_eq!(input, expected_ast, Expr);
}

Now that the parser knows how to build these chains, we can begin to interpret them correctly.

Fixing the treewalk interpreter

The treewalk implementation is closest to my mental model, so I began there. The main trick is to reuse the right-hand side of each evaluation in the next comparison. Once we hit the first False result, we can short-circuit the entire operation. If we got to the end without seeing any False results, we are free to return True.

fn evaluate_comparison_chain(
    &self,
    left: &Expr,
    ops: &[(CompareOp, Expr)],
) -> TreewalkResult<TreewalkValue> {
    let mut left = self.evaluate_expr(left)?;

    for (op, right) in ops {
        let right = self.evaluate_expr(right)?;

        // ideally, we wouldn't need to clone `right` here, but this requires ownership for now.
        let result = self.evaluate_compare_op(left, op, right.clone())?;
        if !result.as_boolean() {
            return Ok(TreewalkValue::Bool(false));
        }

        left = right;
    }

    Ok(TreewalkValue::Bool(true))
}

The other fun part about the treewalk interpreter is that it’s further along, which means I support operator overloading. Each time a comparison chain evaluates a particular operation, under the hood it is calling a dunder method on an object.

fn evaluate_compare_op(
    &self,
    left: TreewalkValue,
    op: &CompareOp,
    right: TreewalkValue,
) -> TreewalkResult<TreewalkValue> {
    use CompareOp::*;

    match op {
        In => self.invoke_method(&left, &Dunder::Contains, args![right]),
        NotIn => {
            let contains = self.invoke_method(&left, &Dunder::Contains, args![right])?;
            Ok(contains.not())
        }
        Equals => self.invoke_method(&left, &Dunder::Eq, args![right]),
        NotEquals => self.invoke_method(&left, &Dunder::Ne, args![right]),
        LessThan => self.invoke_method(&left, &Dunder::Lt, args![right]),
        GreaterThan => self.invoke_method(&left, &Dunder::Gt, args![right]),
        LessThanOrEqual => self.invoke_method(&left, &Dunder::Le, args![right]),
        GreaterThanOrEqual => self.invoke_method(&left, &Dunder::Ge, args![right]),
        Is => Ok(TreewalkValue::Bool(left.is(&right))),
        IsNot => Ok(TreewalkValue::Bool(!left.is(&right))),
    }
}

This part technically already existed, but there’s something satisfying about watching a new feature click into an old mechanism. These comparisons used to live inside my binary operator evaluation, back when CompareOp didn’t exist and comparison chains and I didn’t yet understand each other.

With the treewalk interpreter working smoothly, it was time to bring the same behavior to the bytecode VM.

Fixing the bytecode VM

When I sat down to write this section, I read through my code to begin to explain it. In classic rubber-duck fashion, I realized I’d overcomplicated the whole thing. I now present the simplified pseudocode of how the bytecode looks for operator chaining.

evaluate left
for each (op, right):
    evaluate right
    compare op
    if false: goto end
    if not last iteration: pop true
    else: leave it
end:

Remember how we had to save the right-hand side for the next comparison? That’s what we’re still missing in this pseudocode. The line compare op will pop two operands off the stack and compare them, consuming both operands in the process. If we want the right operand to still be around afterwards, we’re gonna need to intervene.

We can do this by running a DupTop and a RotThree before we do the actual comparison. (CPython uses COPY and SWAP.) The former makes a copy of the right-hand operand, while the latter moves it to the third spot on the stack (hence the rotate three).

Here’s what this looks like in Rust.

fn compile_comparison_chain(
    &mut self,
    left: &Expr,
    ops: &[(CompareOp, Expr)],
) -> CompilerResult<()> {
    if ops.is_empty() {
        panic!("Comparison chain must have >= 1 op.");
    }

    self.compile_expr(left)?;

    let mut false_jumps = vec![];
    for (i, (op, right)) in ops.iter().enumerate() {
        let last_op = i == ops.len() - 1;

        self.compile_expr(right)?;

        // Preserve the right-hand side for the next comparison
        if !last_op {
            self.emit(Opcode::DupTop)?;
            self.emit(Opcode::RotThree)?;
        }

        self.emit(Opcode::from(op))?;

        // If any comparison evaluates to False, jump to end.
        // Otherwise, pop the True and continue the chain.
        // Unless it's the last operation, and we should leave the result on the stack.
        if !last_op {
            // At the end of the loop, we will patch this with a JumpIfFalse
            false_jumps.push(self.emit_placeholder()?);

            self.emit(Opcode::PopTop)?;
        }
    }

    // Patch all fail jumps to here
    for placeholder in false_jumps {
        let offset = self.forward_offset_to(placeholder)?;
        self.emit_at(placeholder, Opcode::JumpIfFalse(offset))?;
    }

    Ok(())
}

The updates to the VM itself were minimal, we just needed to support the new DupTop and RotThree.

    Opcode::DupTop => {
        let x = self.peek()?;
        self.push(x)?;
    }
    Opcode::RotThree => {
        // Before:
        // c <- TOS
        // b
        // a
        //
        // After:
        // b <- TOS
        // a
        // c
        let c = self.pop()?;
        let b = self.pop()?;
        let a = self.pop()?;
        self.push(c)?;
        self.push(a)?;
        self.push(b)?;
    }

My initial implementation only worked when all the operands were equal, like 2 == 2 == 2. That told me I had a bug in the RotThree implementation which I corrected (and left better notes behind for future-me).

Putting it all together

With both interpreter engines updated, it was time to add tests to crosscheck. Here’s a subset of the chaining tests which are run through both engines.

#[test]
fn operator_chaining() {
    let input = "2 == 2 == 2";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 == 2 == 22";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));

    let input = "2 == 2 == 2 < 3";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "1 < 2 < 3 < 4";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "1 < 2 < -3 < 4";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));

    let input = "1 < 2.0 < 3";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 in [1,2,3] in [[1,2,3],[4,5,6]]";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 in [1,2,3] in [[4,5,6]]";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));
}

👉 Curious what these examples compile to? Try it live in the Interactive Bytecode Compiler: fromscratchcode.com/bytecode-compiler

Sometimes I get lazy and/or scared and don’t add the full test suite to crosscheck, but I was determined to do this feature right. (hence the blog post.)

Along the way, I realized I’d implemented 2 < 2.5 but not 2.5 < 3. The first calls Dunder::Lt on int, while the second calls it on float. It’s fun to catch and quickly fix oversights like this. In addition, 2 == 2.0 never returned True. This was because I’d assumed that any cross-type comparison should always return False.

The end

Operator chaining isn’t a flashy feature, yet implementing it reminded me what I love about continuing to develop Memphis: the more Python behaviors I properly model—across two engines—the purer my abstractions become. And what is expertise in software, if not fluency in abstractions? That’s what keeps me coming back to build this interpreter no one asked for.

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

What I’ve learned from 200+ hours helping developers grow

Jones Beach — Mon, 14 Jul 2025 11:30:00 +0000

This isn’t a how-to guide. Just a few things I’ve noticed while mentoring people 1:1 over the last two years.

One-on-one mentorship wasn’t even part of my career plan. I viewed myself as a quiet builder and, hopefully, a good teammate.

This journey started as most things do in my life: an experiment because I was curious. I had an itch for more 1:1 conversation and a hope that I could help a person or two with Python, so I made a profile on Wyzant.

Two years and more than 200 hours later, it’s become one of the most human parts of my work. I’ve worked with college students, self-taught developers, and software professionals, focusing on helping them build confidence and clarity in topics like Python, Rust, React, and even digital logic and computer architecture.

Here’s what I’ve learned.

You don’t need to be extroverted

One of my early fears, both in mentorship and in starting my own business, was that I would fail if I wasn’t “on.” I wasn’t sales-y, and I worried I wouldn’t know how to manage a conversation that got away from me. I was even scared I wouldn’t know how to cut off a session at the end of the hour!

But none of that turned out to matter.

In fact, the structured sessions ended up being exactly what my brain needed. Already knowing someone wants to talk to me when I hop on a video call, even if it is loosely transactional, puts me at ease.

This let me focus on the real work: showing up respectfully, staying curious about what brought someone in, and posing the right question at the right time.

If they’re not asking questions, that might be a good sign

Sometimes I’ll catch myself thinking, Why aren’t they asking me anything? as I watch a mentee explore something on their own.

But I stop myself, because that’s actually a good sign. It means they’ve taken the wheel, and something about our environment still feels useful to them. My challenge then becomes to provide encouragement and nudge them with bigger questions.

A few months ago, one of my mentees was thinking aloud and said “I bet you’re gonna tell me to ask ChatGPT.” I smiled because it showed they had internalized when to use one of their tools. I was still there to help them interpret the next steps, but they were able to take the first step on their own.

Emotional safety looks different for everyone

Our first sessions are often knee-deep in technical detail, but as we zoom out, I try to get a sense of the environment around the work.

How do they like their manager? For the college students, do they have friends in class?

That stuff is often lurking beneath the surface. A lonely or unsupportive environment can be just as challenging as a gnarly stack trace. And I want to make space for both.

Not everyone engages deeply on those questions, but it doesn’t mean they won’t come back. I’ve had consistent mentees where we go deep into mental health issues and others where we never get past the weather. Learning to be okay with that has been part of the work too.

Code is often just the entry point, we stay for the connection

You hear this kind of thing about food or music, but I can’t say I hear many people say, “I code to connect with people.” As a craft, it can be the ultimate solitude. Even when folks talk about using code “for good,” it’s often from a distance. Which is fine!

But for me, it’s more personal. Nearly all of my close friends came from engineering school or work. It was an environment I felt confident in, and that confidence gave me a safe harbor from which to go meet people. Now I try to offer that same sense of support to people who might not have had it elsewhere.

People often reach out to get unstuck technically, which gets us into the space together. But I really enjoy the chance to hear what’s going on in their lives. What’s coming up after graduation, what’s been weighing on them.

I don’t get that deep with everyone I work with, but after three or so sessions we usually have some rapport. My goal is that they leave feeling more seen than when they came in, ideally with their code running.

That’s everything I’ve learned. Nothing else! Zip, zero, zilch.

Oh, and Vite replaced Create React App while I was asleep.

If you’re looking for technical mentorship that includes both the code and the human behind it, I’d love to work with you. Book a free intro call, and let’s get started.

This was cross-posted on From Scratch Press.

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

How global variables work in Python bytecode

Jones Beach — Mon, 16 Jun 2025 11:30:00 +0000

Think globally, act locally. Who knew this oft-touted phrase was referring to Python bytecode?

Last time we acted on learning how local variables work, today let’s think about how global variables might work.

I came into my own bytecode journey assuming that a “global” was no different than a “local,” it was just at the outermost scope of a module. And boy, was I mistaken.

While the VM isn’t even aware of a local variable’s name, just its index, the VM performs dynamic name resolution to resolve each global variable. This is a key part of Python’s dynamism.

Once again, this post will discuss the mechanics of Memphis, my Python interpreter built in Rust. While it doesn’t match CPython 100%, the model is simple and maps closely to how most Python runtimes behave under the hood.

👉 Try Memphis live with my Interactive Bytecode Compiler: fromscratchcode.com/bytecode-compiler

Bytecode Compilation

Consider the following Python program.

# A global variable which we'll later reference inside our function
y = 11

# A function which adds an unknown number (via global var) to the input
# parameter x, returning the result.
def add_useless(x):
    return x + y

Here is the bytecode for our function, add_useless:

LOAD_FAST    0 (x)
LOAD_GLOBAL  0 (y)
ADD
RETURN_VALUE

Our CodeObject looks like the one below. Again, pardon my pseudo-Rust syntax! Or...

👉 Try this example live in my Interactive Bytecode Compiler: fromscratchcode.com/bytecode-compiler

CodeObject {
  name: "add_useless",
  varnames: ["x"], // names of each local accessed by the function
  names: ["y"], // names of each global accessed by the function
}

We don’t see a constant of 11 here because this code object is specific to our add_useless function. The function will have to look up that value at runtime via the global store, since it isn’t embedded in the code object itself.

If you haven’t read the previous post, I’d encourage you to pause here and give it a read. It walks through the evaluation stack step-by-step and shows how the ADD operation gets its two operands off the stack.

VM Execution

What we’re interested in today is what, exactly, LOAD_GLOBAL is doing. First, let’s define a runtime concept I blew past earlier.

Global Store: a mapping from a variable name to an object reference. The object being referenced lives on the heap. This is roughly what you get when you call globals() in Python or CPython.

Let’s see what happens when the VM begins executing the bytecode. Like last time, we’ll ignore how the bytecode actually enters the function.

LOAD_FAST 0 means: read from slot 0 on the frame’s execution stack.
LOAD_GLOBAL 0 means: look up the identifier in names at index 0. It finds y, then goes to the global store to look for the key y, where it finds an object reference to the value of 11.
The remaining instructions, ADD and RETURN_VALUE, work just as they did in the previous post: the VM pops two operands off the stack, computes the result, and returns it.

You may have noticed: global variables don’t need to be defined when the function is defined, only when it is called. If y doesn’t exist at function call time, a NameError will be thrown. This differs from local variables, which must exist at the time they are first referenced. Otherwise, the bytecode compiler will assume you are referring to a global with that same name.

See how there is an extra level of indirection when resolving the global compared to the local? This is key to Python’s dynamism. Here’s an example to illustrate this.

y = 11

def add_useless(x):
    return x + y

print(add_useless(9)) # prints 20

y = 1

print(add_useless(9)) # prints 10

This example highlights the perils of impure functions, but otherwise might not seem that surprising at first.

Before we continue, it’s worth mentioning that the global store is specific to the module, while the heap is shared across modules. This means that multiple modules can have a global named y, which matches what we expect as users. Imagine having to know if every other module in your project, including those from third-party libraries, had previously used a global identifier? That would be a nightmare!

Global Store Mutations

To further illustrate how globals can be modified, let’s consider two more examples. These below move away from my Memphis implementation and focus on how dynamic global behavior shows up in CPython.

a = "First"
globals()['a'] = "Second"
print(a) # prints "Second"

This one accomplishes something similar to the previous example (reassigning a global), but by mutating the global store directly via globals(). We see here that a doesn’t point to a fixed memory location, but is resolved dynamically by name at runtime.

To take it one step further, consider the same idea applied across modules. We’ll also reassign a function rather than an integer.

# main.py
import other

def monkey_patch_foo():
    print("Got ya!")

other.foo = monkey_patch_foo
other.bar() # prints "Got ya!"

# other.py
def foo():
    print("I am foo.")

def bar():
    foo()

Our main module modifies the behavior of a function defined in another module, and that change affects the other module itself. Said another way: you can change the behavior of code from the outside, and the function itself doesn’t know.

This is referred to as monkey patching and is one of the most mind-bending things about Python. Monkey patching is often used to inject patch fixes or replace functionality at runtime. And it’s only possible because of Python’s dynamic name resolution for global variables.

To be transparent, Memphis doesn’t fully support this yet! I'm still working through object mutability and shared references, and how these vary across my treewalk implementation and bytecode VM. Monkey patching: WIP.

You might wonder why Python treats locals and globals so differently in bytecode.

The reason is to balance performance and flexibility. Functions are called frequently during the lifetime of a program, its locals are accessed often, so Python optimizes for speed using slot-based indexing. At the module level, the dynamic name resolution introduces a performance hit, but the language is more dynamic as a result.

The End

I didn’t fully appreciate this design until I implemented it myself. You really start to see how much of Python’s feel comes from how its variables are resolved in bytecode.

If you’re curious to explore more, I recommend trying out the dis module to inspect your own functions! It’ll be confusing at first, but you may begin to pick up one new tidbit each time.

Now that we’ve tackled locals and globals, we’re free to think about free variables next! I hope you’ll stay tuned.

Lastly, I’m continuing to experiment with open office hours this week. If you’re stuck in Python or Rust and want to talk through it live, here’s the booking link. The slots are pay-what-you-want, with zero pressure to tip. I’d love to meet you!

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

How local variables work in Python bytecode

Jones Beach — Mon, 02 Jun 2025 11:30:00 +0000

Local variables are stored on the stack.

This line tormented me for years. I could spout it in an interview, but I didn’t really know what it meant. I could pull up the diagram showing the stack growing upwards and the heap growing downwards. And I definitely didn’t want to think about what happened if they ever crossed.

But to truly understand? I had to build it myself.

This post explains how local variables work by walking through their mechanics in Memphis, my Python interpreter built in Rust. While it doesn’t match CPython byte-for-byte, the model is simple and maps closely to how most Python runtimes behave under the hood.

👉 Try Memphis live with my Interactive Bytecode Compiler: fromscratchcode.com/bytecode-compiler

If you’ve read my previous posts, you’ll recall I have two implementations: a treewalk interpreter and a bytecode VM. This post focuses on the bytecode VM because it had a steeper learning curve, blew my mind more often, and is closer to CPython.

Let’s begin with some definitions which will be used by our bytecode VM at runtime.

Bytecode VM concepts

CodeObject: An immutable structure produced by the bytecode compiler and consumed by the VM.

FunctionObject : A runtime representation of a CodeObject plus its captured environment. We’ll need this layer for when we introduce free variables, which underpin closures.

Frame: A runtime representation of a FunctionObject. It holds the program counter and the evaluation stack, including the space reserved for local variables.

CallStack: A stack of Frame objects, which tracks each execution context (i.e each new function call).

What the heap?!

Wait a minute. Are local variables, which so famously live on the stack, stored in the Frame or the CallStack? The answer is yes!

Each Frame contains its own evaluation stack, the bottom portion of which is reserved for local variables. This is what the term “stack-based virtual machine” refers to, and is in contrast to a register-based VM. Both are mechanisms for managing working memory during your program’s execution.

Meanwhile, the CallStack is one layer of abstraction higher. It holds a new Frame for each new function which is entered.

Before I go any farther, I want to address the elephant in the room. In addition to the incantation local variables are stored on the stack, I was pretty sure that objects in Python are stored on the heap. What gives?

Both things are true. Local variables are stored on the stack, but most variables in Python are references. The stack holds the pointer to the object (which could an int, string, list, custom object, etc.), while the heap does hold the actual data.

Let’s walk through a short function to see this in action.

The stack in action

Consider the following function. It takes in one parameter and defines one additional local variable. For the scope of this discussion, let’s assume our VM has already called our function, add_useless.

# A function which adds an unknown number (via local var) to the input
# parameter x, returning the result.
def add_useless(x):
    y = 11
    return x + y

Remember how we said the bottom portion of the Frame's evaluation stack is reserved for local variables? We can visualize it like this.

┌────────────────┐
|   stack[1]     │
├────────────────┤
|   stack[0]     │
├────────────────┤
│   locals[1]    │
├────────────────┤
│   locals[0]    │
└────────────────┘

Before we set y = 11, the stack would hold two reserved spaces for our two local variables, x and y.

┌──────────────────┐
│ empty slot for y │
├──────────────────┤
│    value of x    │
└──────────────────┘

During the evaluation of y = 11, the stack briefly looks like this:

┌──────────────────┐
│         11       │
├──────────────────┤
│ empty slot for y │
├──────────────────┤
│     value of x   │
└──────────────────┘

After we set y = 11, the stack would be:

┌────────────┐
│    11 (y)  │
├────────────┤
│ value of x │
└────────────┘

During the evaluation of x + y, the stack briefly looks like this:

┌────────────────┐
|       11       │
├────────────────┤
|   value of x   │
├────────────────┤
│      11 (y)    │
├────────────────┤
│   value of x   │
└────────────────┘

After we add x + y, but before the return the stack would be:

┌────────────────┐
│ value of x + y │
├────────────────┤
│      11 (y)    │
├────────────────┤
│   value of x   │
└────────────────┘

Then we return the sum, discard the frame, and hope nothing blows up. (Can you tell I don’t want to explain that part right now?)

If you are screaming at your screen “why is only half of our stack the stack,” I get it! It confused me too that is one piece of contiguous memory, but we reserve the bottom n slots for n local variables.

So how do we know how many local variables to reserve? Because we already compiled the function’s bytecode before running it. Let’s take a look at how that works.

The bytecode in action

Before we look at the bytecode itself, let’s define a few more concepts. These are both fields on the CodeObject and are populated during the bytecode compilation.

constants : The list of constants taken directly from the user code. Think: any integer, float, bool, or string. Because CodeObjects themselves are immutable, they can also appear as constants. In Rust, I represent this with an enum to allow multiple types in the same list.

varnames: The list of identifiers for each local variable in a given CodeObject. In a previous post, I described how varnames (for locals) differs from names (for globals).

During compilation, each identifier assigned to (either as a function parameter or a local variable) is pushed into the varnames list of the current CodeObject. (We can ignore the global keyword for now.)

Whenever the compiler encounters a new function definition, it creates a fresh CodeObject and pushes it onto a code stack. This stack is used only during compilation to track lexical scope, and is not related to the runtime CallStack. Once a function body is fully compiled, the corresponding CodeObject is popped off and made available as a constant for its parent CodeObject.

Here is the bytecode for our add_useless function. The data in the parentheses annotates what each index refers to. I added this manually from my Memphis output, but you can get a similar output in CPython by running import dis; dis.dis(add_useless), which uses the dis module.

LOAD_CONST  0  (11)
STORE_FAST  1  (y)
LOAD_FAST   0  (x)
LOAD_FAST   1  (y)
ADD
RETURN_VALUE

👉 Try this example live in my Interactive Bytecode Compiler: fromscratchcode.com/bytecode-compiler

We’re indexing into two separate lists here, both of which live inside a CodeObject. I write Rust, so naturally I visualized it like this:

CodeObject {
  constants: [11],
  varnames: ["x", "y"],
}

LOAD_CONST indexes into constants, while STORE_FAST and LOAD_FAST index into varnames.

The magic here is there’s no need for the runtime VM to know about the names of the local variables. varnames is used by the compiler to emit the right indices into the bytecode, but during execution, the VM just deals with stack slots.

Once a runtime Frame is initialized, the VM will execute each of these instructions and index into its own execution stack.

The End

If this felt a bit dense, I totally get it! It took me several tries to get this working and clean in my Memphis implementation.

I’m hoping to write similar posts covering global variables and free variables, so I hope you’ll stick around for those.

Lastly, I’m experimenting with open office hours this week. If you’re stuck in Python or Rust and want to talk through it live, here’s the booking link. The slots are pay-what-you-want, with zero pressure to tip. I’d love to meet you!

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

Verifying two interpreter engines with one test suite

Jones Beach — Mon, 05 May 2025 11:30:00 +0000

Crosscheck, my cross-engine testing framework for Memphis, had been sitting atop my writing to-do list for a while.

Instead, I deleted it and replaced it. Memphis can now test itself across all engines.

What does this mean?

I can verify the treewalk interpreter and bytecode VM interpreter produce the same results (return value or symbol table entries) for a given snippet of Python.
This functionality is available in unit tests, rather than only in integration tests. In Rust, this means in src/ rather than tests/.

Let’s look at a couple of examples.

Testing expressions

Any interpreter worth its table salt should be able to evaluate 2 + 2.

[An expression returns a value, whereas a statement modifies control flow or the symbol table. I couldn’t have defined this clearly before I began this project, so I hope this helps.]

#[test]
fn binary_expression() {
    let input = "2 + 2";
    assert_crosscheck_return!(input, MemphisValue::Integer(4));
}

A few building blocks were necessary to get here.

First, I needed to represent a Python value independent of the runtime engine. This isn’t exactly necessary for integers, but come lists or objects, my architecture needed each engine to be able to manage interior mutability differently.

Second, I needed a simple initialization flow, which would hide the complexity of initializing two interpreter engines and running a snippet of code through both. I’d had a few builders over time, but they always did too much. I wanted to truly hide all this behind assert_crosscheck_return!. Speaking of which, let’s look inside.

macro_rules! assert_crosscheck_return {
    ($src:expr, $expected:expr) => {{
        let mut session =
            $crate::crosscheck::CrosscheckSession::new($crate::domain::Source::from_text($src));
        let (tw_val, vm_val) = session.eval();
        assert_eq!(
            tw_val, $expected,
            "Treewalk return value did not match expected"
        );
        assert_eq!(
            vm_val, $expected,
            "Bytecode VM return value did not match expected"
        );
    }};
}

Awesome! It calls something else. Sounds and reads like software.

It’s worth mentioning here why assert_crosscheck_return is a macro rather than a helper function. When a panic happens inside a macro, it reports the line number of your test file, not of the macro itself. This is due to how Rust resolves them before compile-time. Some of my macros call helper functions under the hood, but having the assertions at the macro-level makes debugging way simpler.

We create a CrosscheckSession, which creates two MemphisContext objects. Since this is test-only, I don’t mind loudly failing with expect inside of eval.

pub struct CrosscheckSession {
    treewalk: MemphisContext,
    vm: MemphisContext,
}

impl CrosscheckSession {
    pub fn new(source: Source) -> Self {
        let treewalk = MemphisContext::new(Engine::Treewalk, source.clone());
        let vm = MemphisContext::new(Engine::BytecodeVm, source);
        Self { treewalk, vm }
    }

    /// Run both engines; confirm they return the same value, then return the value. Useful
    /// for evaluating expressions or statements which only return a single value.
    pub fn eval(&mut self) -> (MemphisValue, MemphisValue) {
        let tw_val = self.treewalk.run().expect("Treewalk run failed.");
        let vm_val = self.vm.run().expect("VM run failed.");

        (tw_val, vm_val)
    }
}

Each MemphisContext manages the lexer/parser/interpreter lifetime for a single Engine.

pub struct MemphisContext {
    lexer: Lexer,
    interpreter: Box<dyn Interpreter>,
}

impl MemphisContext {
    pub fn new(engine: Engine, source: Source) -> Self {
        let lexer = Lexer::new(&source);
        let interpreter = init_interpreter(engine, source.clone());

        Self { lexer, interpreter }
    }

    pub fn run(&mut self) -> Result<MemphisValue, MemphisError> {
        let MemphisContext {
            lexer, interpreter, ..
        } = self;

        let mut parser = Parser::new(lexer);
        interpreter.run(&mut parser)
    }
}

Testing statements, classes, and more

Let’s look at a slightly more involved example, a test which defines a class with one attribute and a couple of methods.

#[test]
fn method_call() {
    let mut session = crosscheck_eval!(
        r#"
class Foo:
    def __init__(self, val):
        self.val = val

    def bar(self):
        return self.val

f = Foo(10)
b = f.bar()
"#
    );
    assert_crosscheck_eq!(session, "b", MemphisValue::Integer(10));
}

Our pair of new macros here allows us to start a crosscheck session. Each session will run the provided code snippet through each engine, then stay alive so we can query its symbol table.

macro_rules! crosscheck_eval {
    ($src:expr) => {{
        $crate::crosscheck::CrosscheckSession::new($crate::domain::Source::from_text($src))
            .run()
            .expect("Crosscheck session failed")
    }};
}

macro_rules! assert_crosscheck_eq {
    ($session:expr, $name:expr, $expected:expr) => {{
        let actual = $session.read($name).expect("Symbol not found");
        assert_eq!(actual, $expected);
    }};
}

Let’s look at the two new CrosscheckSession methods, run and read. You’ll notice I break my own rule about keeping assertions at the top level inside the read method. I’ll put a ticket in the backlog to fix that later.

impl CrosscheckSession {
    /// Run both engines; discard the return value and return the session. Useful for 
    /// later reads from the symbol table with `CrosscheckSession::read`.
    pub fn run(mut self) -> Result<Self, MemphisError> {
        self.treewalk.run()?;
        self.vm.run()?;
        Ok(self)
    }

    /// Read a value from both engines; confirm they return the same value, then 
    /// return the value.
    pub fn read(&mut self, name: &str) -> Option<MemphisValue> {
        let a = self.treewalk.read(name)?;
        let b = self.vm.read(name)?;

        assert_eq!(a, b, "Engines returned different values");
        Some(a)
    }
}

And that’s it! I have a few other macros and flows to validate an expected Python runtime error, but they follow the same ideas as what I’ve shown here.

Why did I delete the old `crosscheck`?

Because it sucked.

Okay not really, it did its job for nearly a year! But it required a lot of boilerplate code. It also expected each test file to manually define what engines to verify (using TreewalkAdapter and BytecodeVmAdapter).

Have a look for yourself:

fn run_binary_expression_test<T: InterpreterTest>(mut interpreter: T) {
    let input = "2 + 2";
    match interpreter.evaluate(input) {
        Err(e) => panic!("Interpreter error: {:?}", e),
        Ok(result) => {
            assert_eq!(result, MemphisValue::Integer(4));
        }
    }
}

#[test]
fn test_treewalk_binary_expression() {
    run_binary_expression_test(TreewalkAdapter::new());
}

#[test]
fn test_bytecode_vm_binary_expression() {
    run_binary_expression_test(BytecodeVmAdapter::new());
}

Early-2024-me was pleased with this, but it is heavy.

I wrote the original as an integration test (where tests live in tests/ rather than src/) because I didn’t know what I was doing. Rust produces a separate binary for integration tests, which was unnecessary for verifying cross-engine behavior.

It also meant I was using memphis as a library — interesting on its own (a Python interpreter you can embed in other Rust code! cool!), but not what I needed here.

The End

This work took roughly a year, though I took about 11 months off in the middle.

If the founder of the Single Responsibility Principle reads this, I hope they are proud of me. Their principle served as a useful North Star.

Along the way, I used MemphisContext to support both engines in the REPL. That’s also the reason the Lexer is kept alive but the Parser isn’t—to support streaming input. But that’s a story for another day.

I hope you are well and your code returns the same value every time!

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

I'm embarrassed by how much code I cut from my test suite

Jones Beach — Mon, 31 Mar 2025 11:30:00 +0000

My parser test suite was a house of cards I’ve never been prouder to see collapse. I found a pattern that worked, kept working, and then worked a little too well. Until rust-analyzer couldn't process my file anymore. Best practice says to refactor before your LSP craps out, but alas. Here's how I saved several thousand lines of test code in my Memphis parser.

Designing an expressive parser test suite

I’m going to break all rules of writing and tell this story Benjamin Button style.

We begin with this idyllic test case. Wouldn’t it be lovely to verify our AST in an expressive yet relaxed way? Yes, it is lovely.

#[test]
fn expression() {
    let input = "2 + 3 * (4 - 1)";
    let expected_ast = bin_op!(
        int!(2),
        Add,
        bin_op!(int!(3), Mul, bin_op!(int!(4), Sub, int!(1)))
    );

    assert_ast_eq!(input, expected_ast, Expr);

    let input = "2 // 3";
    let expected_ast = bin_op!(int!(2), IntegerDiv, int!(3));

    assert_ast_eq!(input, expected_ast, Expr);
}

This test wasn’t all sunflowers and rainbows. This 16-line fact-checker used to come in at a bloated 38 lines.

Have a peep yourself.

#[test]
fn expression() {
    let input = "2 + 3 * (4 - 1)";
    let context = init(input);

    let expected_ast = Expr::BinaryOperation {
        left: Box::new(Expr::Integer(2)),
        op: BinOp::Add,
        right: Box::new(Expr::BinaryOperation {
            left: Box::new(Expr::Integer(3)),
            op: BinOp::Mul,
            right: Box::new(Expr::BinaryOperation {
                left: Box::new(Expr::Integer(4)),
                op: BinOp::Sub,
                right: Box::new(Expr::Integer(1)),
            }),
        }),
    };

    match context.parse_oneshot::<Expr>() {
        Err(e) => panic!("Parser error: {:?}", e),
        Ok(ast) => assert_eq!(ast, expected_ast),
    }

    let input = "2 // 3";
    let context = init(input);

    let expected_ast = Expr::BinaryOperation {
        left: Box::new(Expr::Integer(2)),
        op: BinOp::IntegerDiv,
        right: Box::new(Expr::Integer(3)),
    };

    match context.parse_oneshot::<Expr>() {
        Err(e) => panic!("Parser error: {:?}", e),
        Ok(ast) => assert_eq!(ast, expected_ast),
    }
}

The tool I used to reduce my boilerplate was declarative macros, Rust’s way of generating Rust code at compile time.

By the end of my cleanup trance, I improved these areas:

expressing Python types (int, str, bool, list, set, tuple)
expressing operations (binary, unary, and logical ops)
wrapping the actual entrypoint to parse the input
wrapping error handling for the common case

The result? Shorter tests and clearer intentions.

Expressing Python types

I can now write int!(3) instead of Expr::Integer(3). Blah blah blah so what.

This one doesn’t save a whole lot, but let’s look at two more.

I can now write list![int!(1), int!(2), int!(3)] and set![int!(1), int!(2), int!(3)]. Glancing at the implementation for those macros, we see another key.

macro_rules! list {
    ($($expr:expr),* $(,)?) => {
        Expr::List(vec![
            $($expr),*
        ])
    };
}

macro_rules! set {
    ($($expr:expr),* $(,)?) => {
        Expr::Set(HashSet::from([
            $($expr),*
        ]))
    };
}

My parser tests no longer care that an Expr::List accepts a Vec, while a Expr::Set accepts a HashSet. One could argue I don’t need the HashSet at all because this is just the AST, not the evaluation stage of the interpreter. In that case, I could change the underlying representation by updating the macro.

Expressing operations

This one starts to get really fun. I can now write bin_op!(var!("a"), BitwiseAnd, var!("b")), which expands to the following.

macro_rules! bin_op {
    ($left:expr, $op:ident, $right:expr) => {
        Expr::BinaryOperation {
            left: Box::new($left),
            op: BinOp::$op,
            right: Box::new($right),
        }
    };
}

Not only does this macro hide the two Box initializations (necessary in a recursive enum), it also allows us to write BitwiseAnd rather than BinOp::BitwiseAnd, all without sacrificing any type checking.

Wrapping the parser entrypoint

Previously, I had to write this, which isn’t awful, but is also awful.

let context = init("2 + 3");

match context.parse_oneshot::<Expr>() {
   ...
}

We’re working with a MemphisContext object here, another bloated structure I use to orchestrate the whole evaluation flow. The problem is, this is an evolving interface. I learned the hard way that without a level of indirection in my tests, I’d have to tweak this pattern constantly. Across hundreds of tests, that is obnoxious.

The new approach uses a straight forward parse! macro.

let ast = parse!($input, Statement);
assert_stmt_eq!(ast, $expected);

Wrapping the happy path error handling

As Yogi Berra famously said, 90% of unit tests are one-half mental.

I applied this philosophy to design parse! to handle the happy path, the roughly 90% of my parser tests I expect to be able to parse their Python input successfully. Since these are tests and nothing matters, we can fail loudly on any unexpected exceptions and return the AST quietly otherwise.

macro_rules! parse {
    ($input:expr, $pattern:ident) => {
        match init($input).parse_oneshot::<$pattern>() {
            Err(e) => panic!("Parser error: {:?}", e),
            Ok(ast) => ast,
        }
    };
}

And for those 10% of tests where we expect a parse error? This will do just fine.

macro_rules! expect_error {
    ($input:expr, $pattern:ident) => {
        match init($input).parse_oneshot::<$pattern>() {
            Ok(_) => panic!("Expected a ParserError!"),
            Err(e) => e,
        }
    };
}

This is what it now looks like to confirm an improperly structured dict. While I love match statements, I love even more no longer having to write them.

let input = "{ 2, **second }";
let e = expect_error!(input, Expr);
assert_eq!(e, ParserError::SyntaxError);

The End

I’m embarrassed I didn’t do these steps sooner. Please learn from my mistakes and treat your tests the way you wish to be treated.

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

I left corporate and still do roadmaps + a Memphis update

Jones Beach — Mon, 03 Mar 2025 12:30:00 +0000

Happy March!

This month I am putting a bow on my Q1 roadmap and continuing to unify the two Memphis execution engines.

I chose an update post because I want to “build in public.” What follows is what has been on my mind! I don’t write advice posts (Why you shouldn’t use Vec) or random technical overviews (Vectors vs. Slices in Rust: A Complete Guide) because I am unable to do either with a straight face. If I could, I’d probably still work in a large organization where those kind of ambitious but safe norms are politely celebrated. Someone else will write those pieces and I support them in the same way I support someone running a marathon; I’m not against it, but that doesn’t mean I want to.

It appears this “build in public” includes me breaking the fourth wall on my writing process! Can you tell I struggle in groups? If you’re still here, let’s continue.

Q1 Roadmap

I began Q1 with a list of 6 items I wanted to achieve for From Scratch. As someone who once flew to Chicago to write a list of personal goals on a public library whiteboard with a close friend, I’m no stranger to making goals. What felt new was doing them with that uniquely American fuel: a profit motive.

The Original 6 (the little-known prequel to The Magnificent 7) were:

Launch website testimonials (thanks Jakub!)
Improve SEO for fromscratchcode.com
Launch lead magnet
Run a small trial with paid ads
Write chapters 3-5 of my novella
File taxes and pay Q1 estimated taxes

I would later add two more:

Add CTA to and modernize my personal site
Create a runbook for follow-ups

I’m still working on Chapter 5 (Chapters 1-4 are on From Scratch Press) and the follow-up runbook, but everything else is DONE. Who knew you could accomplish things without Jira for Enterprise.

These records have unintentionally produced a fairly detailed account of the playbook I’m running to build my online business. Which I’m excited to share with you in case you too are interested in how to earn small amounts of money with only an internet connection.

I feel pride looking at this list because it makes my efforts feel less random. I can reassure myself Sure, I’m still growing, but here’s my strategy. I can (and have!) thrown this list into ChatGPT and asked what foundational pieces am I missing. And each time it says “Share your work on social media,” I close the tab and search for From Scratch on Google instead.

Multiple Execution Engines

Memphis has supported two execution engines for about a year now. But barely.

The treewalk interpreter is farthest along and, if you wanted to try to run real code, what you should use. I’ve been strengthening the bytecode VM’s foundation and it’s coming along but slowly. Because I have no deadlines, I’m being deliberate to define Memphis capabilities versus treewalk capabilities versus bytecode VM capabilities. Did I mention I have no deadlines?

The unification has proceeded in the following broad strokes.

Common Entrypoint

Initially, you could pick an engine like this.

# Treewalk is default FOR NOW
memphis example.py

# VM selected using an environment variable
MEMPHIS_ENGINE=vm memphis example.py

Which works fine! But after kicking off the runtime, there was no coming back together. Or reunification.

This remains the interface to select a non-default engine, but I’m gradually unifying more of the code behind the scenes. I’ve also floated the idea of using a Rust feature flag instead of an environment variable to produce a smaller binary.

Common Return Type

Because I respect those who came before me, I wanted to treat Python runtime errors as first-class. Meaning I wouldn’t trap them below deck after hitting an iceberg.

Instead of separate error types for treewalk and VM errors, I would combine them. This would also be a chance to turn this dumping ground I aspirationally called InterpreterError into a type which actually represented possible Python runtime errors.

// The treewalk version was first so it is known as "Interpreter" here
pub enum InterpreterError {
    Exception(DebugCallStack),
    TypeError(Option<String>, DebugCallStack),
    KeyError(String, DebugCallStack),
    ValueError(String, DebugCallStack),
    NameError(String, DebugCallStack),
    AttributeError(String, String, DebugCallStack),
    FunctionNotFound(String, DebugCallStack),
    MethodNotFound(String, DebugCallStack),
    ClassNotFound(String, DebugCallStack),
    ModuleNotFound(String, DebugCallStack),
    DivisionByZero(String, DebugCallStack),
    ExpectedVariable(DebugCallStack),
    ExpectedString(DebugCallStack),
    ExpectedInteger(DebugCallStack),
    ExpectedList(DebugCallStack),
    ExpectedTuple(DebugCallStack),
    ExpectedRange(DebugCallStack),
    ExpectedSet(DebugCallStack),
    ExpectedDict(DebugCallStack),
    ExpectedFloatingPoint(DebugCallStack),
    ExpectedBoolean(DebugCallStack),
    ExpectedObject(DebugCallStack),
    ExpectedClass(DebugCallStack),
    ExpectedFunction(DebugCallStack),
    ExpectedIterable(DebugCallStack),
    ExpectedCoroutine(DebugCallStack),
    WrongNumberOfArguments(usize, usize, DebugCallStack),
    StopIteration(DebugCallStack),
    AssertionError(DebugCallStack),
    MissingContextManagerProtocol(DebugCallStack),
    RuntimeError,
    EncounteredReturn(ExprResult),
    EncounteredRaise,
    EncounteredAwait,
    EncounteredSleep,
    EncounteredBreak,
    EncounteredContinue,
}

pub enum VmError {
    StackUnderflow,
    StackOverflow,
    NameError(String),
    RuntimeError,
}

I landed on this streamlined structure which could now be used on both engines. All my previous ExpectedString, ExpectedInteger, etc., variants are now just a TypeError. This mirrors CPython, where an optional String parameter provides more detail.

pub struct ExecutionError {
    pub debug_call_stack: DebugCallStack,
    pub execution_error_kind: ExecutionErrorKind,
}

pub enum ExecutionErrorKind {
    RuntimeError,
    ImportError(String),
    TypeError(Option<String>),
    KeyError(String),
    ValueError(String),
    NameError(String),
    AttributeError(String, String),
    DivisionByZero(String),
    StopIteration,
    AssertionError,
    MissingContextManagerProtocol,
}

With this unified type, I was able to test for an expected NameError in a crosscheck test. Which reminds me, I should really write more about crosscheck, my testing framework for both engines. I have a fun proc macro in the works there.

Now that both engines returned an ExecutionError, I needed to build an interface to push new stack frames to the DebugCallStack. These would be displayed to the user whenever a Python runtime error occurs in their user code.

Common Debug Stack Trace

I’m still actively working on unifying stack traces, but I’m excited because it is me saying “Memphis supports a debug stack trace, regardless of what execution engine you choose,” rather than just “both engines have a stack trace.” Do you see the difference? It’s a pLaTfOrM. Or maybe I mean fRaMeWoRk? It’s something.

I cleaned up my DebugStackTrace and DebugCallStack structs and moved them into a domain module to indicate they represent a Python stack trace, but should NOT be used as a source of runtime info for an engine.

One challenge is giving each engine access to the right-sized shared state object. This would be how each engine could register new stack frames; think something like state.push_stack_frame(function.to_stack_frame()) when entering a new function context. I’m attempting to balance platform capabilities (MemphisState) against freedom of implementation within each engine (TreewalkState and VmState).

The other challenge of implementing stack traces is it forces you to keep around your metadata for use at the right time. Metadata like file paths and line numbers aren’t necessary for actually running Python code, but they’re essential for debugging. When a statement is parsed and immediately evaluated (treewalk) or immediately compiled (bytecode VM, though this has a bug right now), we record the line number of the start of the current function and increment it each time we see a new line. This parser<>runtime communication is only necessary for stack traces (for me. so far.).

This work stream has been a crash course in applying the Single-Responsibility Principle a few decades after I first learned of it. It’s easy to understand the definition (”of course each function/struct/class only does one thing!”). But applying it? That’s harder (”we’ve got access to state here so I’ll put it there!”). The result is my code is GRADUALLY shifting from a medium number of medium-sized functions to a whole lot of tiny ones. I’ve always been an advocate for adding a second entrypoint, either through unit tests or a REPL, but Memphis has forced me to expand that thinking to an entirely different level.

The End

I’m putting the finishing touches on my Q2 roadmap, which should take my business to the max! Do people still say that?

I recently finished The Pathless Path by Paul Millerd and his message of finding the work each of us wants to do indefinitely resonated with me. With my Memphis engine work and my roadmap work, I believe I’m closer than ever to finding that. I’d also love to be a technical mentor to anyone reading this. Because money.

Hope you are well!

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

Build a software career with meaning: a playbook

Jones Beach — Mon, 17 Feb 2025 12:30:00 +0000

Even with 20 fingers and toes, I barely have enough digits to count how many times during my 9-5 career I thought, “Wow, I feel frustrated/stuck/alone/hungry, maybe I should look for a new job.” A new job holds the allure of allowing you to let go of all your negative emotions (especially hunger) and start over.

There are plenty of reasons to switch jobs, particularly when advocating for fairer compensation or a less-toxic environment. But if you're like me and looking to build a meaningful software career, where you can use your brain to its fullest and maybe do a bit of good, job hopping is more like a salve on a wound than a complete fix.

With this in mind, I’m excited to announce my new email course: Build a Software Career with Meaning. While under development, I called this “How I went from ‘Hello World!’ to ‘How can I help?’ in just 19 years,” which is clearly a cheeky title but captures the work-in-progress feel of my career.

Did I mention it’s free? Over the course of 5 (business) days, you’ll receive a brief email with a piece of wisdom, along with an actionable thought experiment you can test on your career.

I also want to be transparent: in email marketing speak, this is called a “lead magnet.” I learned this term from reading (does YouTube still exist?), so I can only assume this is referring to the element from the periodic table with the symbol Pb (you know, the one pronounced ‘led’). And here I thought lead wasn't magnetic!

I'm offering the course for free because I want to share how I think about things. At the end of 5 (business) days, perhaps you’ll feel like you know a bit more about my values and want to work with me. If you don’t, that’s more than okay! You are welcome to stay on the list indefinitely, and the option to unsubscribe will always be at the bottom. There are no tricks here, just a marketing playbook I’ve been learning in between implementing my Markdown blog and nested functions in Python bytecode.

If you’ve ever felt stuck or disillusioned in your software career, I hope this course gives you a new perspective. If nothing else, it’ll be five (business) days of me popping into your email client with career advice just a tad more nuanced than “Quit your job!”

P.S. If you’re interested in reading the story about how my 9-5 career crashed-and-burned—and how I built something better from the wreckage—I have the complete account over on From Scratch Press. I’d love to hear if any of my experiences and catatonic thought loops (the kind where you forget to eat) mirror your own!

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

Building a Markdown blog with links optimized for Gatsby

Jones Beach — Mon, 03 Feb 2025 12:30:00 +0000

While I was hoping this website would build itself, it ended up taking a good amount of fine-tuning! I wanted to share how I built this blog and a few of the challenges I encountered.

Background

My setup for this website is:

domain and DNS managed on GoDaddy
code lives in a private GitHub repository (should I make this public?)
static site written in React using Gatsby
site deployed using the free tier on Netlify

While Dev.to and Hashnode have been helpful to get me flowing on my technical writing, I really wanted a minimal blog over which I exercised full control. I wanted to be able style blockquotes the way I liked. I wanted the blog to reflect the From Scratch ethos that less is better.

The final straw: when I realized I was missing out on potential SEO improvements by not publishing my writing directly to fromscratchcode.com.

Thus this silly blog that I’m moderately proud of was born!

List of Requirements

My requirements for the new blog were:

Posts shall be written in Markdown and be portable, meaning require no changes to be thrown into other editors (Dev.to, Hashnode, etc). As a bonus, I currently write these in Notion because they copy out already in Markdown.
Posts shall support syntax highlighting for code blocks that actually look good. (This was a PAIN on my email provider. too bad this post isn’t about picking an email provider!)
Internal links within posts shall be optimized for navigation and SEO.

Gatsby is modern, flexible, and I’d barely used it before, which gave me the confidence I could pull this off. I wrote this post because these requirements got more difficult as I went and I hope this can be a resource for someone else. Here’s what I learned!

Supporting Markdown Posts

My motivation for writing in Markdown will not surprise you: I wanted to spend more time writing and less time formatting. To move a piece (including code!) between platforms and have it just work. Markdown checks all of those boxes.

Gatsby’s ecosystem is rich with markdown support. Here’s how I leveraged it!

I used the gatsby-transformer-remark plugin to read the Markdown files. Next, I used the createPages API in gatsby-node.js to register a new page with slug (this is a URL that for some reason is named after an insect).

Then, I used Gatsby’s createNodeField to attach additional metadata to the Markdown node. This ensures this metadata is available to components via GraphQL queries. I also used the reading-time library to calculate the reading time for each block of Markdown to give each post some fun deets and make this look like a real blog.

Here are these pieces assembled together in my gatsby-node.js file:

const BLOG_QUERY = `
    {
      allMarkdownRemark(
        filter: { fileAbsolutePath: { regex: "/blog/" } }
        sort: { frontmatter: { date: DESC } }
      ) {
        edges {
          node {
            frontmatter {
              slug
            }
          }
        }
      }
    }
  `

// This function runs a GraphQL query and creates pages at `${base_url}/${node.frontmatter.slug}`
// for each using the provided template.
// NOTE: each markdown file must provide its own slug in the frontmatter.
// TODO: could we provide a default in the case the frontmatter is not specified?
const createMarkdownPages = async (
  graphql,
  createPage,
  query,
  base_url,
  template
) => {
  const markdownFiles = await graphql(query)

  markdownFiles.data.allMarkdownRemark.edges.forEach(({ node }) => {
    const slug = `${base_url}/${node.frontmatter.slug}`
    createPage({
      path: slug,
      component: template,
      // The context is needed for the $slug param lookup in the query inside the
      // repsective template
      context: {
        slug,
      },
    })
  })
}

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions
  const blogPostTemplate = path.resolve(`src/templates/blogPostTemplate.js`)

  await createMarkdownPages(
    graphql,
    createPage,
    BLOG_QUERY,
    "/blog",
    blogPostTemplate
  )
}

exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions
  if (node.internal.type === "MarkdownRemark") {
    let slug = node.frontmatter.slug

    // Determine the base path (e.g., 'policies' or 'blog')
    // This is the name from the gatsby-source-filesystem config in gatsby-config.js
    const sourceInstanceName = getNode(node.parent).sourceInstanceName

    // Add the slug field to the node, which will become queryable from GraphQL. This is safer
    // than relying on frontmatter within the component pages because we have applied all the
    // necessary transformations here before we write to the field.
    createNodeField({
      node,
      name: "slug",
      value: `/${sourceInstanceName}/${slug}`,
    })

    // Calculate the reading time of the markdown content and add it to the GraphQL
    const stats = readingTime(node.rawMarkdownBody)
    createNodeField({
      node,
      name: "readingTime",
      value: stats.text, // Example: "3 min read"
    })
  }
}

I trimmed for brevity (HA!), but I process my Terms of Use and Privacy Policy using the same procedure. While those would not need syntax highlighting, requirement #3 about optimized internal links would still apply.

Next, I used Gatsby’s GraphQL to query for the post content based on the slug of the rendered page.

This is my first pass at blogPostTemplate.js. I have stripped out a few pieces to focus on the GraphQL query.

import React from "react"
import { graphql } from "gatsby"

const BlogPost = ({ data }) => {
  const post = data.markdownRemark

  return (
    <Layout>
      <article>
        <h1>{post.frontmatter.title}</h1>
        <div dangerouslySetInnerHTML={{ __html: post.html }} />
      </article>
    </Layout>
  )
}
export const query = graphql`
  query ($slug: String!) {
    markdownRemark(fields: { slug: { eq: $slug } }) {
      frontmatter {
        title
        date(formatString: "MMM DD, YYYY")
      }
      fields {
        readingTime
      }
      html
    }
  }
`

export default BlogPost

Consider the following piece of frontmatter, the metadata at the top of a Markdown file:

---
title: "Introducing: From Scratch Code"
date: "2024-11-04"
slug: "introducing-from-scratch-code"
---

At this point, we have created and populated a page at /blog/introducing-from-scratch-code/. Awesome!

Enabling Syntax Highlighting

Syntax highlighting ended up taking two tries, both using PrismJS.

First Attempt: `gatsby-remark-prismjs`

My first approach used the gatsby-remark-prismjs plugin (yes, this is a plugin [prismjs] to a plugin [remark] to a framework [gatsby]) during the Gatsby build pipeline. Here was the addition to my gatsby-config.js file.

{
  resolve: `gatsby-transformer-remark`,
  options: {
    plugins: [
      {
        resolve: `gatsby-remark-prismjs`,
        options: {
          classPrefix: "language-", // Set a class prefix
          inlineCodeMarker: null, // Marker for inline code
          // Do not use prism to highlight inline code
          noInlineHighlight: true,
        },
      },
    ],
  }
}

This approach does these steps behind the scenes:

Read markdown
Convert it to HTML
Apply syntax highlighting via CSS classes
Let React render our HTML which we fetched by querying GraphQL

This worked out of the box!

To customize the theme, I added this to my gatsby-browser.js.

// Prism.js syntax highlighting
import "prismjs/themes/prism-tomorrow.css"

I had some trouble styling line numbers and the Copy-to-Clipboard button. There seemed to be some complexity around PrismJS plugins inside a static site processing pipeline such as Remark in Gatsby, so I punted on those.

This approached worked great until it didn’t, which brings me to requirement #3.

Optimizing Internal Links

When I say “internal link”, I mean a link on this website, such as /blog.

I had two motivations for optimizing these:

This is mostly me being particular, but in order to truly make my Markdown portable, I wanted to be able to write external links (such as https://fromscratchcode.com/mentorship) in Markdown and have it be treated as the internal link /mentorship.
This is the biggie: Gatsby applies a magic touch using its Link component, which consists of several things.
1. Under the hood, Gatsby prefetches any Link destination when the page loads.
2. When clicked, it uses @reach/router to navigate to it using ultra-smooth client-side navigation.
3. This is all while appearing to be an <a> component in the page source, which keeps these links *chef's kiss* perfect for SEO because they remain visible to crawlers such as Google’s.
4. In addition, our React state persists across clicks. Without this optimization, the state of the dark mode toggle would persist when using the top nav, but not when clicking a link discovered in a blog post. That is not ideal!

Second Attempt: Switching to Rehype

The challenge here is turning [mentorship](/mentorship) into <Link to="/mentorship">mentorship</Link> and have it still be rendered as React.

There is a strong ecosystem of JS libraries to help with this, but I needed to switch from using PrismJS in a Remark plugin to a Rehype plugin. That sentence would have been word salad to me a few ~~hours~~ weeks ago, so please bear with me. Gatsby’s Remark pipeline converts Markdown to HTML before React sees it, so internal links get set before React can modify them. By using Rehype, we get a hook where we can dynamically swap <a> for Link. Rehype also supports integrating PrismJS syntax highlighting. Here’s how I pulled it off!

The function below does this whole shebang in several steps:

Parse our raw markdown content into an AST (Abstract Syntax Tree).
Apply a plugin we would need to write to detect and convert external links to internal links.
Convert the markdown AST (used by Remark) into an HTML AST (used by Rehype).
Apply PrismJS syntax highlighting using a Rehype plugin.
Render the Rehype AST as React, converting any <a> elements into CustomLink components along the way. (CustomLink is a wrapper I created around Gatsby’s Link so that it works for internal or external links.)

// Render markdown to React
// This pipeline processes raw markdown and converts it into React components,
// applying syntax highlighting and internal link optimization.
const renderMarkdownToReact = markdown => {
  try {
    return unified()
      .use(remarkParse) // Parse markdown into an abstract syntax tree
      .use(optimizeInternalLinks) // Apply our plugin to detect internal links
      .use(remarkRehype) // Convert Markdown AST to Rehype AST
      .use(rehypePrism, { showLineNumbers: false }) // Add PrismJS syntax highlighting
      .use(rehypeReact, {
        jsx,
        jsxs,
        Fragment,
        components: {
          a: props => <CustomLink to={props.href} {...props} />,
        },
      })
      .processSync(markdown).result
  } catch (error) {
    console.error("Failed to render markdown:", error)
    return <div>Error rendering markdown</div>
  }
}

And our plugin to convert internal links:

const optimizeInternalLinks = () => tree => {
  visit(tree, "link", node => {
    const { url } = node
    if (url.startsWith("/") || url.startsWith(SITE_URL)) {
      const internalPath = url.replace(SITE_URL, "")
      node.url = internalPath
    }
  })
}

This function calls visit from unist-util-visit to allow us to walk the AST. Its interface matches what is expected by a Remark plugin.

PHEW. That is quite a pipeline. This is the type of problem I probably would have given up on pre-ChatGPT. With a tool that points me to exactly what libraries to use and gets me close on the initial syntax, I was able to take it the rest of the way.

Now you can click around this blog and, when I reference a past post, it should load nearly instantaneously and preserve your dark mode settings. We did it!

What’s next?

I’m pleased with how the blog has turned out! I’d like to eventually add support for tags, table of contents for each post, and perhaps a series so to link to all the Memphis posts. I could add comments using Disqus but, then again, would you invite a troll into your living room?

I’d love to hear from you! Please reach out if you find any bugs. I’m also curious what static site blog features have wow-ed you in the past, now that I am the proud owner of my own.

I’m gonna go write some Rust now.

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

Typed integers in Rust for safer Python bytecode compilation

Jones Beach — Mon, 13 Jan 2025 12:30:00 +0000

Shortly after I shared my previous post, a helpful redditor pointed out that the typed integers I alluded to is known as the newtype pattern in Rust, a design that allows you to define types purely for compile-time safety and with no runtime hit.

And here I thought I invented it! Alas. 😂

I wanted to share a few details about what challenge I encountered and how I solved it.

The problem

Like I mentioned in the previous post, bytecode is a lower-level representation of your code created for efficient evaluation. One of the optimizations is that variable names are converted into indices during bytecode compilation, which supports faster lookup at runtime when the VM pumps through your bytecode.

At one point while implementing this myself, I was debugging a sequence that felt like this. (I’m using the wishy-washy “felt like” here because I have no idea what the actual bytecode looked like.)

LOAD_GLOBAL 0
LOAD_FAST 0
LOAD_CONST 0

Okay, so we’re loading the first global, the first local, and the first constant. Given the ongoing evolution of my VM design and implementation, the data structures I’m using to communicate the mappings of these indices to their symbol names and/or values between the compiler and VM have been in a steady flux.

I’d sit down to implement the execution of a given opcode in the VM, see that I was handed the value 0, and I’d think, “What do they want me to do with this?!” (“they” in the case being myself from 5 minutes prior hacking on the compiler stage.)

After interpreting a 0 wrong for the forth or fifth time, I decided THERE HAS TO BE A BETTER WAY. I asked myself, “Is there a way to get the compiler to tell me when I’m using a 0 incorrectly?” In addition to compile-time checking, with rust-analyzer configured in my Neovim, I should get a type error as soon as I made the mistake.

Could I add types to my integers?!

The CPython bytecode docs even hint at these distinctions, but incorporating them into my own implementation wasn’t immediately clear until I experienced the pain firsthand. For my three opcodes above, the respective documentation is:

LOAD_GLOBAL(namei)
Loads the global named co_names[namei>>1] onto the stack.

LOAD_FAST(var_num)
Pushes a reference to the local co_varnames[var_num] onto the stack.

LOAD_CONST(consti)
Pushes co_consts[consti] onto the stack.

co_names, co_varnames, and co_consts are three fields on a CPython code object, which is an immutable piece of compiled bytecode. In my interpreter, I’m using names and varnames to show my originality.

(Note to self: I treat constants separately at the moments, but I should probably unify it as I solidify my understanding of code objects.)

(Another note to self: how curious that they are shifting namei to the right during LOAD_GLOBAL. I hope to one day know why.)

The solution

I added typed integers to keep myself sane. They offered an additional benefit of providing me an opportunity to gain more experience with generics in Rust!

Here is a subset of my Opcode implementation.

pub enum Opcode {
    /// Push the value found at the specified index in the constant pool onto the stack.
    LoadConst(ConstantIndex),
    /// Read the local variable indicated by the specified index and push the value onto the stack.
    LoadFast(LocalIndex),
    /// Read the global variable indicated by the specified index and push the value onto the stack.
    LoadGlobal(NonlocalIndex),
}

My hope here is these new types, ConstantIndex, LocalIndex, and NonlocalIndex, would semantically illustrate the behavior of each opcode while providing type safety.

Generics entered the scene next as I was hoping to implement this with minimal code reuse.

Here is the next layer of the onion.

pub type ConstantIndex = Index<ConstantMarker>;
pub type LocalIndex = Index<LocalMarker>;
pub type NonlocalIndex = Index<NonlocalMarker>;

We’re beginning to see some code reuse–awesome!

What are these marker types? They are truly that: a type which is literally just a marker with no other data. These empty types are enforced at compile-time and do nothing at runtime.

pub struct ConstantMarker;
pub struct LocalMarker;
pub struct NonlocalMarker;

In Rust, PhantomData<T> from std::marker allows you to include a type in a struct purely for compile-time checks without affecting runtime performance—exactly the type safety for which we’re seeking! I’m using usize for the value because my use-case was indices, which should never be negative.

/// An unsigned integer wrapper which provides type safety. This is particularly useful when
/// dealing with indices used across the bytecode compiler and the VM as common integer values such
/// as 0, 1, etc, can be interpreted many different ways.
#[derive(Copy, Clone, PartialEq, Hash, Eq)]
pub struct Index<T> {
    value: usize,
    _marker: PhantomData<T>,
}

impl<T> Index<T> {
    pub fn new(value: usize) -> Self {
        Self {
            value,
            _marker: PhantomData,
        }
    }
}

impl<T> Deref for Index<T> {
    type Target = usize;

    fn deref(&self) -> &Self::Target {
        &self.value
    }
}

impl<T> Display for Index<T> {
    fn fmt(&self, f: &mut Formatter) -> Result<(), Error> {
        write!(f, "{}", self.value)
    }
}

The last piece of magic is impl<T> Deref for Index<T>, which allows us to treat instances of our new type as integers when dereferenced.

As a result, a piece of code like this would pass with flying colors.

#[test]
fn test_dereference() {
    let index: LocalIndex = Index::new(4);
    assert_eq!(*index, 4)
}

By this point, we are free to use these in the bytecode compiler! Most places in the code don’t require specifying the generic because it will be inferred by the function signature, like in this example.

fn get_or_set_local_index(&mut self, name: &str) -> LocalIndex {
    if let Some(index) = self.get_local_index(name) {
        index
    } else {
        let code = self.ensure_code_object_mut();
        let new_index = code.varnames.len();
        code.varnames.push(name.to_string());
        Index::new(new_index)
    }
}

We initialize a new index with Index::new(new_index), which the compiler knows to treat as an Index<LocalMarker>, which we have aliased to LocalIndex to allow us to be blissfully ignorant VM developers.

The end

This is a small but powerful example of how I’m falling head-over-heels for Rust’s type system. I love writing expressive code to implement core features from scratch in a way which manages complexity and makes people smile.

My mentor at my first big-boy job was a wizard at this and I credit him for showing me what is possible: spending more time thinking about what you’re trying to build rather than being bogged down by how you are trying to build it.

Have you used Rust’s type system in any fun and creative ways? Or done something similar in another language? I’d love to hear your thoughts in the comments. Be well!

Subscribe & Save [on nothing]

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

DEV Community: Jones Beach

Write your first library

What is a library?

You’ll feel more confident in your stack.

Your discernment around other libraries will increase.

You’ll learn to design interfaces.

It’s empowering.

Memphis goes online: adding socket support

A net module for Memphis

Implementing the net module

1. Building the module

2. Defining our native structs

3. Wiring them into the type system

4. Adding builtin functions

5. Storing native payloads on objects

Introducing Enid

Wrapping up

Python’s operator chaining is mildly interesting

Fixing the parser

Fixing the treewalk interpreter

Fixing the bytecode VM

Putting it all together

The end

What I’ve learned from 200+ hours helping developers grow

You don’t need to be extroverted

If they’re not asking questions, that might be a good sign

Emotional safety looks different for everyone

Code is often just the entry point, we stay for the connection

How global variables work in Python bytecode

Bytecode Compilation

VM Execution

Global Store Mutations

The End

How local variables work in Python bytecode

Bytecode VM concepts

What the heap?!

The stack in action

The bytecode in action

The End

Verifying two interpreter engines with one test suite

Testing expressions

Testing statements, classes, and more

Why did I delete the old crosscheck?

The End

I'm embarrassed by how much code I cut from my test suite

Designing an expressive parser test suite

Expressing Python types

Expressing operations

Wrapping the parser entrypoint

Wrapping the happy path error handling

The End

I left corporate and still do roadmaps + a Memphis update

Q1 Roadmap

Multiple Execution Engines

Common Entrypoint

Common Return Type

Common Debug Stack Trace

The End

Build a software career with meaning: a playbook

Building a Markdown blog with links optimized for Gatsby

Background

List of Requirements

Supporting Markdown Posts

Enabling Syntax Highlighting

First Attempt: gatsby-remark-prismjs

Optimizing Internal Links

Second Attempt: Switching to Rehype

What’s next?

Typed integers in Rust for safer Python bytecode compilation

The problem

The solution

The end

A `net` module for Memphis

Implementing the `net` module

Why did I delete the old `crosscheck`?

First Attempt: `gatsby-remark-prismjs`