DEV Community: Aleksander Obuchowski

MedImageInsight: Open-Source Medical Image Embedding Model - now on HuggingFace

Aleksander Obuchowski — Mon, 04 Nov 2024 19:13:50 +0000

TLDR: check out the model at https://huggingface.co/lion-ai/MedImageInsights

Making Medical Image AI Actually Accessible

If you've ever tried implementing a medical imaging model, you know the drill: promising papers, complicated setup processes, and documentation that assumes you are a certified Zzure developer. When I came across the MedImageInsight model, it was the same story - great potential buried under layers of enterprise infrastructure.

It took me 6 hours to access the model - first I had to register on azure, set up payment organization etc. Then although the code repository was there there was no option to clone or download the repo so I had to manually copy the content of each file! Not to mention that this model was shared as MLFlow artifact so there was a ton of unnecessary code. But, since the model is shared on MIT licence I decided to share my custom implementation on Huggingface so you don't have to through the same hell as I did

What's MedImageInsight Anyway?

At its core, MedImageInsight is a dual-encoder model (think CLIP, but for medical images) that can:

Convert medical images into meaningful embeddings
Match images with text descriptions
Perform zero-shot classification on medical conditions
Handle multiple medical imaging modalities (X-rays, CT scans, etc.)

The model was trained on a massive dataset of medical images and their descriptions, learning to create a shared embedding space for both images and text. This means you can throw new medical conditions at it without retraining, and it'll do a decent job at identifying them.

Why Another Implementation?

The original implementation required:

An Azure account
MLflow setup
Multiple enterprise-level configurations
Dealing with undocumented dependencies
Coffee. Lots of coffee.

After spending way too much time setting it up, I decided to strip it down to its essentials. No shade to the original authors - they created an amazing model. But not everyone needs enterprise-grade MLflow pipelines to run a few predictions.

How It Actually Works

At its heart, MedImageInsight uses a technique called contrastive learning to create a shared understanding between medical images and their descriptions. Think of it as teaching the model to speak two languages fluently: the language of images and the language of medical terminology.

The Power of Zero-Shot Learning

Traditional machine learning models are like students who can only answer questions they've seen before. Zero-shot learning models, on the other hand, are like students who can apply their knowledge to entirely new situations.

MedImageInsight achieves this through a clever architectural design:

One part of the model learns to understand medical images
Another part learns to understand medical terminology
Both parts are trained to translate their understanding into the same "language" (a shared vector space)

This means if you show the model a chest X-ray and ask "Is there pneumonia?", it doesn't need to have seen pneumonia examples during training. Instead, it understands both what pneumonia means textually and what to look for in the image.

Getting Started

Clone the repo:

git clone https://huggingface.co/lion-ai/MedImageInsights

Install dependencies (we use uv because it's fast and deterministic):

uv sync

Run the example:

uv run example.py

That's it. No Azure setup, no MLflow, no enterprise infrastructure required.

What's Next?

We're working on:

Better explainability (what is the model actually looking at?)
HuggingFace's transformers library compatibility
More example notebooks for specific use cases
Performance optimizations

Contributing

Found a bug? Have an improvement in mind? The repository is actually open source (imagine that!), and we welcome contributions.

Resources

Original paper: arXiv:2410.06542
Code: https://huggingface.co/lion-ai/MedImageInsights
Example notebooks: In the examples directory
FastAPI service: In fastapi_app.py

How to make your first chat bot in 50 lines of code — theory and practice

Aleksander Obuchowski — Sun, 19 Apr 2020 18:15:13 +0000

How to make your first chat bot in 50 lines of code — theory and practice

1. How chat bots work?

Most of modern chat bot platforms consist of 3 main
things — intent recognition ,slot filling and dialog graph.

1.1 Intent recognition

Intent recognition is a text classification task which goal is to
capture specific intent behind a user query. This is motivated by the fact that
users tend to formulate their request in a lot of different ways so we need to
have a system that is able to tell if those messages relate to the same thing or
not. Let’s illustrate this with an example of a bank chat bot, where users can
ask the bot to withdraw money:

Visualization of intent detection system

You can see that although the requests are formulated in a lot of different ways
and in different styles they all mean basically the same thing and chat bot
should react in the same way. Therefore we need **text classification **model
that captures the semantics behind user sentences and assigns them to the
specific predefined class.

1.2 Slot filling

Once we know what action the user wants to take we need to capture specific
parameters of those actions. For example, if you want Alexa to play your
favorite song, you want her to play this specific song not just any song, so
besides detecting intent chat bots also need to
perform a task that is called slot-filling.

1.3 Dialog graph

Another requirement for chat bot functionality is dialog graph. It’s goal is to steer conversation in the right direction. For example when you say “Check the weather” the chat bot could then ask “What day should I check the weather for?” and next it will be looking for intents like ‘tomorrow’ or ‘today’. The important part here is, there would be no point in asking the second question without the first one, so there is a need for a system that stores the information of the point in the conversation where we are and what are the possible next states.

1.4 Our chat bot

In this tutorial our goal is to create a simple chat bot, so we are going to focus only on intent detection task and simple dialog graph model. This is enough to make a chat bot that is able to answer FAQ and conduct as simple conversation.

2. Word and sentence embedding

Our goal in designing an intent detection is to create a system that, given a few examples for intent, can detect that a sentence given by the user is similar to these examples and therefore should have the same intent.

The problem behind this system is that we have to design a system for checking if 2 sentences are similar. This could be achieved by eg. counting how many overlapping words are in the new sentence and the sentences in training dataset. This is however a naive approach because a user can use a word that has similar meaning, but is different from the ones in the train examples.

2.1 Word embedding

A solution here is to use word embedding.

Word vectors ref : https://ruder.io/word-embeddings-1/](https://ruder.io/word-embeddings-1/

Word embeddings are mathematical representations of words encoded as vectors in n-dimentional space. Similar (used in the same context) words are close to each other in this space. This means that we can compare 2 or more words to each other not by e.g. the number of overlapping characters but by how close they are to each other in they embedded form.

2.2 Sentence embeddings

From word embeddings we can construct embeddings for the whole sentence. This can be done in a variety of ways, we can simply take the average of the word vectors, use weighted average to check how important the words are by e.g tf-idf
coefficient or even use more advanced methods like transformer neural
networks.

2.3 Similarity

Once we have prepared embeddings for the sentences we have to design a way for comparing them. A simple widely used method here is cosine similarity that measures similarity between two vectors as the angle between them.

Cosine similarity ref: https://bit.ly/2X5470I

3.Building the chat bot

To create the sentence embedding we are going to use flair library. This library is based not only on static word embeddings but also analyses the words character by character which helps in dealing with out-of-vocabulary words.

In our model we are going to embed the examples for each intent and then, while processing the users message, find the most similar one. This approach is mainly taken as fast and simple one, illustrating how embedding work. Most of modern systems use neural networks (link to related articles can be found at the end),however this approach can still be used if you want to design a system that is fast and and doesn’t use a lot of resources.

We begin our program with creating the outline of the model.

outline of the program

Description

1–9: importing necessary libraries
11 : initialization of the flair model for creating embeddings of sentences. We are using English word embeddings and mean polling method for creating sentence embeddings from word embeddings.
13–20 : chatbot class, this class has two static methods one for creating embeddings and one for processing user message and answering it.

3.1 Preparing embeddings

Firstly we need to prepare a file containing our intents and their examples.This is a json dictionary that uses intents as keys and tables of examples as values.

intents.json:

Next we need to to create a function that constructs embeddings for the
examples.

Description:

4 : Creating new python dictionary for the embeddings
5–6: Opening the input file and loading it to python dictionary 7–8 : For each intent we create a table in the embeddings dictionary
9–12 For each example in the intent, we create a Flair sentence object that we can later embed using the model specified earlier. Finally we add the embedded sentence to the table
13–14: If the file doesn’t exist, we create it
15: We save the embedded dict. We use pickle instead of json to store the numpy arrays

3.2 Answering the message
answers.json:

Description

3: We use the embeddings model
4 -5 : We load load the embeddings file created earlier
6–8 : Embedding of user message
9–10 :Initializing best intent and best sore variables
11–16 For each intent we loop through it’s embedded examples and check the cosine similarity between users message and those examples. We chose the intent, which example has the highest similarity with the new message
17–18 : Loading the answers dict
19: Checking if intent chosen by the system is in the answers dict
20 : Return random answer from the ones assigned to the chosen
intent

Whole code

4. Possible improvements

In this format the chat bot has to choose one of
the intents provided. This means we have no way of detecting if user said
something that doesn't belong to any of the intents. A possible solution is to check the numerical values of the cosine similarity and based on those observation assign a threshold value below which the bot will classify the message as the one it doesn’t know how to answer.