Memphis goes online: adding socket support

Before we begin, let’s agree that “online” in this context means accepting bytes from localhost. Deal? Deal.

You may recall that sometime last century I had the brilliant idea to boot a Flask server from inside Memphis. Trudging toward that goal exposed me to a ton of types and functionality I didn’t yet support. I even ran an interpreter inside an interpreter! But it was a slog. Literally line 1 imports something from the stdlib which imports something from the stdlib which imports something else from the stdlib. I never got to line 2. At a certain point, I realized there was no light at the end of the tunnel.

Skip to a few weeks ago, it hit me I could make my whole life easier by writing my own HTTP library in Python, wire up the necessary system resources from the Rust side, and start running some curl commands.

This post documents Memphis’ journey from being air-gapped to responding in kind:

> curl localhost:8080
Hello from Enid (powered by Memphis)

We’ll approach this like this:

1. What we want a net module for Memphis to look like
2. How I implemented this
3. Introducing Enid, a micro HTTP framework to wrap this up (and allow us to pretend we got Flask working all along)

A net module for Memphis

The surface area to respond to an HTTP request is surprisingly small. (Maybe that’s why I keep reimplementing them? Let’s put a pin in that for later.)

Essentially, we need to:

1. Listen on a port
2. Accept new connections on that port

That’s it!

Here’s a minimal interface to do this.

from memphis.net import listen

sock = listen(("127.0.0.1", 8080))
print("Listening...")
conn, _ = sock.accept()
data = conn.recv(1024)
print("Sending...")
conn.send(b"HTTP/1.1 200 OK\r\n\r\nHello from Memphis!")
print("Closing...")
conn.close()

If you want to skip ahead, here’s a short video showing this in action.

This code should block on the sock.accept(), just like any socket-based server, until it receives a connection on localhost:8080.

In other words, this is the simplest possible Memphis web server, one that says hello over TCP. And we format the bytes we send back as HTTP so we can test this with curl.

You’ll notice we’re importing memphis.net.listen rather than socket like you would in CPython. That’s what we have to build. From scratch!

To do so, we’ll need:

• a memphis.net module, which contains
• a listen function, which, when called, returns
• a Socket object, which contains an accept method which, when called, returns
• a Connection object, which contains recv, send, and close methods.

That’s a lot to wire up! But we can split this into two parts:

• the wiring up of that laundry list
• connecting the Socket and Connection objects to use Rust utilities to actually do things

This is the paradox of implementing a programming language: you spend 90% of your time on the piping, and then just call your host language to do the real work.

Here our tools are Rust’s TcpListener and TcpStream. Let’s take a look at how we can connect those to our Python code.

Implementing the net module

1. Building the module

Our first step is to create a memphis.net module, which will contain the symbols listen, Socket, and Connection.

fn builtins() -> Vec<Box<dyn CloneableCallable>> {
    vec![Box::new(NetListenBuiltin)]
}

fn init(type_registry: &TypeRegistry) -> Module {
    let mut net_mod = Module::new(Source::default());
    for builtin in builtins() {
        net_mod.insert(&builtin.name(), TreewalkValue::BuiltinFunction(builtin));
    }

    register_native_class::<Socket>(&mut net_mod, "Socket", type_registry);
    register_native_class::<Connection>(&mut net_mod, "Connection", type_registry);

    net_mod
}

pub fn import(module_store: &mut ModuleStore, type_registry: &TypeRegistry) {
    let net_mod = init(type_registry);

    let memphis_mod = module_store.get_or_create_module(&ImportPath::from("memphis"));
    memphis_mod.borrow_mut().insert(
        "net",
        TreewalkValue::Module(Container::new(net_mod.clone())),
    );

    module_store.store_module(&ImportPath::from("memphis.net"), Container::new(net_mod));
}

We store our new module in the ModuleStore, which will allow us to find our module when a user’s code imports it.

Next, let’s take a look at the two native structs that back the classes we just registered.

2. Defining our native structs

First, here is our native Socket, which is a thin Rust wrapper.

use std::{
    io,
    net::{SocketAddr, TcpListener, TcpStream},
};

pub struct Socket {
    listener: TcpListener,
}

impl Socket {
    pub fn new(host: String, port: usize) -> io::Result<Self> {
        Ok(Self {
            listener: TcpListener::bind(format!("{}:{}", host, port))?,
        })
    }

    pub fn accept(&self) -> io::Result<(TcpStream, SocketAddr)> {
        self.listener.accept()
    }
}

And our native Connection, another thin Rust wrapper.

use std::{
    io::{self, Read, Write},
    net::{Shutdown, TcpStream},
};

pub struct Connection {
    stream: TcpStream,
}

impl Connection {
    pub fn new(stream: TcpStream) -> Self {
        Self { stream }
    }

    pub fn recv(&mut self, bufsize: usize) -> io::Result<Vec<u8>> {
        let mut buffer = vec![0; bufsize];
        let n = self.stream.read(&mut buffer)?;
        Ok(buffer[..n].to_vec())
    }

    pub fn send(&mut self, data: &[u8]) -> io::Result<()> {
        self.stream.write_all(data)
    }

    pub fn close(&mut self) -> io::Result<()> {
        self.stream.shutdown(Shutdown::Both)
    }
}

It’s worth mentioning that these are both synchronous. I could integrate these with tokio down the road, but that felt like a can of worms best left for future me.

These structs encapsulate the socket behaviors we need, but we still need to connect those up to their Python bindings. You’ll notice these two structs don’t reference our actual interpreter logic: no Module, no knowledge they will be used to evaluate Python code. That’s on purpose!

To make them useful, we’ll bridge our native structs to our Python type system.

3. Wiring them into the type system

In the snippet below, we register our builtin methods to Socket and Connection. impl_method_provider is a macro to reduce the boilerplate to assign our methods to a type.

impl_method_provider!(Socket, [AcceptBuiltin]);
impl_method_provider!(Connection, [ConnRecv, ConnSend, ConnClose,]);

fn register_native_class<T: MethodProvider>(
    mod_: &mut Module,
    name: &str,
    type_registry: &TypeRegistry,
) {
    let object_class = type_registry.get_type_class(&Type::Object);
    let type_class = type_registry.get_type_class(&Type::Type);

    let mut class = Class::new_direct(name, Some(type_class.clone()), vec![object_class.clone()]);

    for builtin in T::get_methods() {
        class.set_on_class(&builtin.name(), TreewalkValue::BuiltinMethod(builtin));
    }

    mod_.insert(name, TreewalkValue::Class(Container::new(class)));
}

In register_native_class, we create a Python Class for each and add these two new classes to our new memphis.net module, each with the specified builtin methods.

With this in place, we can finally begin filling out our builtin methods, starting with NetListenBuiltin.

4. Adding builtin functions

In Memphis, all builtin functions or methods are just structs which implement the Callable trait. This trait requires the struct provide its name and implement a call method. The struct itself doesn’t need any fields because those will all be passed in as runtime parameters.

When a builtin is invoked, it’s provided a reference to the interpreter &TreewalkInterpreter and any arguments Args. The former is needed to access any other runtime resources or raise errors, the latter because that’s how functions work.

pub struct NetListenBuiltin;

impl Callable for NetListenBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        check_args(&args, |len| len == 1, interpreter)?;

        let host_port = args.get_arg(0).as_tuple().raise(interpreter)?;
        let host = host_port.first().as_str().raise(interpreter)?;
        let port = host_port.second().as_int().raise(interpreter)?;

        let socket = Socket::new(host, port as usize)
            .map_err(|e| interpreter.runtime_error_with(format!("Failed to bind Socket: {}", e)))?;

        let socket_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Socket"))
            .ok_or_else(|| interpreter.runtime_error_with("Socket class not found"))?;

        let obj = Object::with_payload(socket_class.clone(), socket);

        Ok(TreewalkValue::Object(Container::new(obj)))
    }

    fn name(&self) -> String {
        "listen".into()
    }
}

In this builtin, we:

• evaluate and type check the positional arguments
• create a new Socket
• look up the memphis.net.Socket class
• create an Object of this class and provide it a Socket

For this last step to work, we must add native object support to our Python Object. Let’s take a look at that now.

5. Storing native payloads on objects

We start by adding a new field to our Object, an optional Box<dyn Any>. I considered using an enum here, something like Option<NativeResource>, but I was stubborn and went with dynamic dispatch. I’ll explain why shortly.

pub struct Object {
    class: Container<Class>,
    scope: Scope,
    native_payload: Option<Box<dyn Any>>, // this is new!!!!!
}

We are now up to the sock.accept() call.

struct AcceptBuiltin;

impl Callable for AcceptBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        let self_val = args.get_self().raise(interpreter)?;
        let socket = self_val.as_native_object::<Socket>().raise(interpreter)?;

        let (stream, addr) = socket.accept().map_err(|e| {
            interpreter.runtime_error_with(format!("Socket.accept() failed: {}", e))
        })?;

        let conn = Connection::new(stream);

        let conn_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Connection"))
            .ok_or_else(|| interpreter.runtime_error_with("Connection class not found"))?;

        let conn_obj = Object::with_payload(conn_class.clone(), conn);

        Ok(TreewalkValue::Tuple(Tuple::new(vec![
            TreewalkValue::Object(Container::new(conn_obj)),
            TreewalkValue::Str(Str::new(&addr.to_string())),
        ])))
    }

    fn name(&self) -> String {
        "accept".into()
    }
}

The key line here is as_native_object::<Socket>(), which reads our new Object.native_payload field and attempts to downcast it to a Socket object.

If I had gone with an enum, I could have used an as_socket() pattern or similar, but I just didn’t want to. There are other potential benefits to using Box<dyn Any>, like dynamic registration of native types, but that feels a ways off.

I won’t show the recv, send, and close methods on Connection, but they work very similarly to accept.

With that, our memphis.net module is wired up and implemented. Thanks for sticking with me! I know those code snippets were a slog.

To recap, we:

• created a new memphis.net module
• populated the module with a listen builtin function and Socket and Connection classes
• added native object support to our Object, with the ability to downcast to retrieve the native objects
• created a native Socket and Connection using TcpListener and TcpStream in Rust
• created builtins for Socket.accept, Connection.recv, Connection.send, and Connection.close

Before we stop, let’s make our HTTP server a bit more fun.

Introducing Enid

With the net module hooked up and working, I wanted to make my HTTP calls look more like Flask, with its decorator usage and general clean feel. I chose to call this Enid, because that’s another small town in the middle of the US.

Our new API looks like this.

from enid import App, Response

app = App()

@app.get("/")
def home(req):
    return Response.text("Hello from Enid (powered by Memphis)")

if __name__ == "__main__":
    app.run()

Any calls to the root / should return a 200 with a text response, everything else should return a 404. Here’s an example curl session.

> curl localhost:8080
Hello from Enid (powered by Memphis)
> curl localhost:8080/
Hello from Enid (powered by Memphis)
> curl localhost:8080/another_endpoint
404 Not Found
> curl localhost:8080/anything_else
404 Not Found
> curl localhost:8080
Hello from Enid (powered by Memphis)

Wrapping up

Implementing socket support also revealed several Python details I’d missed. I didn’t support __name__, I had a ROUGH kwargs bug, plus I needed to implement str.encode, str.join, str.split, and bytes.decode.

If you want to try it on GitHub, you can run an Enid app on either Python or Memphis. With ChatGPT’s assistance, I wrote a shim so that it will try to import from Memphis, then fallback to Python’s stdlib, like so:

try:
    from memphis.net import listen
except ImportError:
    # Fallback to the pure-Python shim
    from .shim_net import listen

This whole project took about two weeks, which is massively shorter than if I’d kept on the road to Flask nirvana. The wiring to reach the inside of the listen function took about a day, getting the rest of the stdlib and pipes working about a week, and the last week was integrating it into the type system.

To show why the type system work was necessary, this snippet shows two things I wasn’t able to do even when I could technically respond to a curl command.

from memphis import net

type(net.Connection) # <class 'type'>
"send" in dir(net.Connection) # True

This reflection was only possible once we registered the Connection class into memphis.net and assigned it methods. This, plus adding the native payload to a Python object, was just as rewarding as seeing the curl work because it meant Memphis is growing up. And while I can’t say this was easy, I also don’t feel like the code is all going to fall apart tomorrow. So we proceed!

There’s a ton here I could do next. Maybe I’ll support route pattern matching (/endpoint/{id}) and middleware in Enid, or get this all working on my bytecode VM. I could even try to wrap my head around what an async version of this would look like. The order I tackle these will probably depend on how I feel when I wake up tomorrow.

With that, Memphis is officially online! Just don’t count on it to respond to your Slack messages.

---

Subscribe & Save [on nothing]

Want a software career that actually feels meaningful? I wrote a free 5-day email course on honing your craft, aligning your work with your values, and building for yourself. Or just not hating your job! Get it here.

Build [With Me]

I mentor software engineers to navigate technical challenges and career growth in a supportive, sometimes silly environment. If you’re interested, you can explore my mentorship programs.

Elsewhere [From Scratch]

I also write essays and fiction about neurodivergence, meaningful work, and building a life that fits. My novella Lake-Effect Coffee is a workplace satire about burnout, friendship, and a coffee van. Read the first chapter or grab the ebook.