Pituophis: Setting up a Gopher server

dotcomboom — Sat, 09 Mar 2019 02:47:11 +0000

This is a tutorial for the Gopher library I wrote called Pituophis, and is current as of version 0.95. While the API is more or less finalized, it may still be subject to change.

Setting up a server

Installing Pituophis

If you haven't already, install Pituophis by running pip3 install pituophis, or the equivalent for your Python 3 setup. You will most likely need Python 3.7 or later, because of its usage of asynchronous socket connections.

The base script

First, you'll need to crack open whatever comfortable environment you have and create a new Python file. Save it to a new directory, and then create a new one named "pub" inside of the same directory.

This is what should go in the file:

import pituophis

pituophis.serve('127.0.0.1', 70, pub_dir='pub/')

For a Gopher server that will serve files from the pub directory, that's it! Pituophis will also send the gophermap file in each directory if it's available. To run the server just save and then start the script. If you're using a port that is below 1024 (as port 70 is) you may need to run it with elevated privileges .

The host, which is 127.0.0.1 in the example, should be changed to whatever host clients will be connecting on. If you're only going to use the server on your machine, it's fine to leave it that, but if you're running it on a private or public IP address or domain name change it to that.

Alt handlers

Pituophis also lets you use an alt handler, a function that will take in requests and then spit out what to send back. If Pituophis, for example, notices that a path does not exist in the publish directory, it will hand off the request to the alt handler.

To use an alt handler, first define it and set it to be used:

import pituophis

def alt(request):
    return False

pituophis.serve('127.0.0.1', 70, pub_dir='pub/', alt_handler=alt)

The alt handler can return False if it can't do anything with the request, so Pituophis will send back an error as usual.

This example uses the alt handler to send back a menu displaying the request's attributes and a couple selectors:

import pituophis

def alt(request):
    if (request.path == '/stats'):
        menu = [
            pituophis.Selector(text="Path: " + request.path),
            pituophis.Selector(text="Query: " + request.query),
            pituophis.Selector(text="Host: " + request.host),
            pituophis.Selector(text="Port: " + str(request.port)),
            pituophis.Selector(text="Client: " + request.client),
            pituophis.Selector(),
            pituophis.Selector(itype="I", text="View server.png", path="/server.png", host=request.host, port=request.port),
            pituophis.Selector(itype="0", text="View some text", path="/txt", host=request.host, port=request.port)
        ]
        return menu
    return False

pituophis.serve('127.0.0.1', 70, pub_dir='pub/', alt_handler=alt)

Remember, if a directory or file called stats exists in the publish directory, the alt handler will not run, and that static file will be sent instead.

The Gopher Protocol in Brief

dotcomboom — Fri, 08 Mar 2019 22:57:45 +0000

Gopher is a protocol for fetching information that dates back to 1993. It's often regarded as a predecessor of sorts to the HTTP protocol that serves as the backbone of the Web as we know it today, however its simplicity and structure gives it a lot of worth.

Gopher is a very simple protocol to work with, and one lunch break could net you basically all you need to know about the protocol. Let me give a rundown.

I'll use (NL) in this article to refer to a newline. According to RFC 1436, this is the equivalent of \r\n, or a Windows/DOS newline. Note that Gopher selectors use tab characters (\t) instead of spaces to separate fields, so you'll need to replace them with tabs in the examples.

Request

The client firsts make a connection to a remote host (a quite necessary step to be frank).

Once this connection is established, the client sends text to the server with the path it wants to access, if applicable, a query string separated by a tab, and followed then a newline, like this:
/fruits pineapple(NL)

Alternatively, the client can just send a newline, and the server will treat it as listing root /.

Response

This is all the information the server needs to process the request. The server then sends over raw data, sometimes ending with a newline followed by a period character (as described in RFC 1436). It then closes the connection.

Unlike HTTP, which uses headers for server responses, Gopher leaves it up to the client to interpret it; this is why a Gopher URL should have an item type in it so the client knows what it will receive.s That brings us to...

URL Schemas

So, that means, instead of this to receive a text document:
gopher://gopher.floodgap.com/gopher/proxy

You should use this, to explicitly tell the client that it'll receive plain text:
gopher://gopher.floodgap.com/0/gopher/proxy

Other (older?) implementations of the Gopher URL are like this, where the item type and path are not separated by a slash:
gopher://gopher.floodgap.com/0gopher/proxy

While this is how it is laid out in RFC 4266, "The gopher URI Scheme", I am not in favor of it. Reason being is how using the malformed URL from earlier would tell the client that it is to fetch opher/proxy and treat it as a g (which happens to be the GIF) item type. Having the slash after the type would make it easier for the client to tell if a URL doesn't have an item type specified, since all types are one character.

Directories/Menus

Gopher servers can send whatever files and raw data to the client, but what ties it all together are menus. Here's an example menu response, playing off of the fruits example from earlier.

1Back to ThingSearch    /   thingsearch.uh  70
iHere are your results for "pineapple". /   error.host  0

Each line in a Gopher menu is a selector, which leads to a resource on a specific server. This can be the same host and port, or another server. Each field is separated by a tab, with the exception of the selector's display text:

The type, which tells the client how to understand the resource when it is fetched from the server.
The display text, without any tab characters. The general rule of thumb is that it should be formatted for a screen of 67 characters across at max. This is not necessarily for technical reasons, however.
The path where the resource is located. In information selectors, this can be left blank.
The host of the server.
The port of the server, which is typically 70.
An extra field could be used to signify that the resource is Gopher+-enabled, although clients that don't support it should be able to ignore it. This isn't shown in the example.

For the menu to be parsed correctly, all of these fields must be present, except the extra field for Gopher+, which is optional.

I've separated a sample of the types that can be used in a menu into four basic categories.

Item types

Text:

0: Plain text

Menus:

1: Directory/Menu
7: Search service (Directory/Menu that accepts a query)

Binaries:

4: HQX file (old Mac archives)
5: PC binary
9: Generic binary (your best bet)
g: GIF image
I: Generic image (be careful when parsing, Gopher item types are case-sensitive)
s: WAV sound (for MP3s and other formats, you should use type 9)

Information:

3: An error
i: Informational text

The 3 and i selectors, even being just information, should still have all of the above required fields filled in. (The path can be blank so long as the tabs are there.) The example above uses a fake host and port. It could be anything, doesn't matter all too much. This way, the few clients that don't interpret information selectors as such will at least have somewhere fake to link to.

Gophermaps

A Gophermap is a file in a directory representing a Gopher menu used by many modern Gopher servers (Gophernicus, Pygopherd, and of course Bucktooth that introduced it). Gophermaps also have a simplified syntax, where the server acts as a preprocessor and will fill in fields and take care of things like relative directories as needed. Additionally, lines without tabs are turned into information selectors automatically.

In a Gophermap, if you're linking to another resource on the same server, you could have a selector that's like this:
1Food food/
and the server will take that and blow it up into a full Gopher selector, filling in the blanks:
1Food /things/food 127.0.0.1 70
In this example, the Gophermap is in the directory /things, the server fills that into the path so the client can understand it. The server knows that it is being advertised on host 127.0.0.1 and port 70 from its configuration, so it fills that in.

Here's another example with some information:
The quick brown fox jumps over the lazy dog.
Because it doesn't have any tab characters, this is turned into something along the lines of:
iThe quick brown fox jumps over the lazy dog. / error.host(T)0

In essence, when using a Gophermap file, the server does the heavy lifting so the client doesn't need to make assumptions when parsing the final Gopher menu.

External links

Common Gopher servers that use Gophermaps:

Bucktooth ^(proxy) (Perl, introduced the Gophermap format)
Pygopherd (Python 2.x)
Gophernicus (C)

These are a couple of my related projects:

Gophermap Editor (HTML5)
Pituophis (Python 3 client/server library)

DEV Community: dotcomboom