Indeterminacy of the `mime` Package in Go: Trust But Verify

Posted originally in my blog: https://lazy.bearblog.dev/go-mime-wtf/

Background

One weekend, I decided to write a Telegram bot for myself to automate to-do or shopping lists.

The idea is simple. I send the bot a list of items to buy, and it creates a webpage with checkboxes that I can tick off while holding my phone in one hand and a shopping basket in the other.

The bot was written pretty quickly, but then I wanted to experiment with OpenAI's Whisper model. So, I decided to add speech recognition to my application.

Now, the program works as follows:

• I send the bot a voice message like this: "buy a kilogram of potatoes, a bunch of dill, a head of cabbage, a couple of beers, oh, don't need the cabbage, we have it"
• The bot recognizes the speech and creates a page with checkboxes:

[ ] potatoes - 1 kg
[ ] dill - 1 bunch
[ ] beer - 2 bottles

Now, about the article. In go-openai, you can send audio using the openai.AudioRequest structure, which allows you to specify an audio data stream in the Reader field or a file path in the FilePath field. Well, I thought these fields were mutually exclusive, but it turns out you need to specify FilePath even when using an audio stream. The API probably uses this to determine the stream type.

On the Telegram side, we receive a tgbotapi.Voice structure, which has a MimeType field. For voice messages, it is audio/ogg, but it could easily change in the future.

So, we have the MIME type of the audio stream, and we need to determine the file extension from it, creating a file name to ensure the OpenAI API doesn't complain about an unknown file type.

What Could Be Simpler?

The code seems trivial:

extensions, err := mime.ExtensionsByType(mimeType)
if err != nil {
    return err
}
if len(extensions) == 0 {
    return fmt.Errorf("unsupported mime type: %s", mimeType)
}
ext := extensions[0]
log.Printf("Assumed extension: %s", ext)
fakeFileName := fmt.Sprintf("voice_message.%s", ext)

Testing locally - it works:

2024/07/28 17:49:06 Assumed extension: oga

Deploying to the cloud - it doesn't work:

2024/07/28 17:55:32 unsupported mime type: audio/ogg

Checking the Go versions, locally and in the CI/CD used for building - they are the same.

Let's Cut to the Chase

It turns out the behavior of the mime package is not deterministic and depends at runtime on whether there is a /etc/mime.types file in the operating system and its contents.

That's where the mime package reads the runtime table of MIME type to file extension mappings.

I'm dead serious: you compile a binary, run all the tests, everything seems great, but this binary will determine the assumed file extensions for the same MIME type differently in different environments.

How to Fix It

Let's take known MIME types of audio files and their extensions, and manually add them in the init function using mime.AddExtensionType.

var mimetypes = [][]string{
    {"audio/amr", "amr"},
    {"audio/amr-wb", "awb"},
    {"audio/annodex", "axa"},
    {"audio/basic", "au", "snd"},
    {"audio/csound", "csd", "orc", "sco"},
    {"audio/flac", "flac"},
    {"audio/midi", "mid", "midi", "kar"},
    {"audio/mpeg", "mpga", "mpega", "mp2", "mp3", "m4a"},
    {"audio/mpegurl", "m3u"},
    {"audio/ogg", "oga", "ogg", "opus", "spx"},
    {"audio/prs.sid", "sid"},
    {"audio/x-aiff", "aif", "aiff", "aifc"},
    {"audio/x-gsm", "gsm"},
    {"audio/x-mpegurl", "m3u"},
    {"audio/x-ms-wma", "wma"},
    {"audio/x-ms-wax", "wax"},
    {"audio/x-pn-realaudio", "ra", "rm", "ram"},
    {"audio/x-realaudio", "ra"},
    {"audio/x-scpls", "pls"},
    {"audio/x-sd2", "sd2"},
    {"audio/x-wav", "wav"},
}

func init() {
    log.Println("init mimetypes")
    for _, v := range mimetypes {
        typ := v[0]
        extensions := v[1:]

        for _, ext := range extensions {
            err := mime.AddExtensionType("."+ext, typ)
            if err != nil {
                log.Fatalf("mime: %s", err.Error())
            }
        }
    }
}

This will make the behavior of our application deterministic regarding the domain it will work with (in my case, audio files).

Conclusion

This experience showed that things in Go are not always deterministic. If something seems odd, don't hesitate to dive into the library's source code and see how it's implemented. This can save you a lot of time and trouble in the long run.