DEV Community: Renato Banzai

Using FHIR to Interact with Natural Language

Renato Banzai — Sun, 29 Aug 2021 19:14:18 +0000

Whats NLP Stands For?

NLP stands for Natural Language Processing which is a field of Artificial Intelligence with a lot of complexity and
techniques to in short words "understand what are you talking about".

And FHIR is...???

FHIR stands for Fast Healthcare Interoperability Resources and is a standard to data structures for healthcare. There are
some good articles here explainig better how FHIR interact with Intersystems IRIS.

My Solution

Chatbot to query FHIR

The most common approach in chatbots is use machine learning to train the model based in old conversations. But when the
chatbot need to use "live" data the challenge is improved. If you already know how a machine learning model works, you know
that is most of cases the trained model is a kind of "hadcorde" application with all decision embedded inside. If you need
more data without retraining the model you will need to create by yourself the integrations and put your mind to work part
in machine learning and part in software engineering.

First Approach

To show more about FHIR I have decided to first work on questions that a healthcare professional could do to a chatbot.
And to do this I keep the software engineering simple, using just javascript and regular expressions to look for intentions
and entities on the chat.

Regular Expressions

This is my favorite subject in processing texts. Regex are a powerful way to work text and match patterns. With regular
expressions I can search for what patient the person chatting wants to know about and what question or chart he wants to see.

Creating a Chatbot with IRIS and Python

Renato Banzai — Sun, 20 Sep 2020 02:04:58 +0000

Creating a Chatbot with IRIS and Python

In this article I'm going to show how to integrate the InterSystems IRIS Database with Python to serve a Machine
Learning Model of Natural Language Processing (NLP).

Why Python?

With a large adoption and use in the world, Python have a great community and a lot of accelerators | libraries to deploy any kind of application.
If you are curious (https://www.python.org/about/apps/).

Iris Globals

As I start to learn about ^globals they became familiar to use as a fast way to ingest data in a out-of-box data model.
So at first I'm going to use ^globals to store training data and the conversations to log the chatbot behaviour.

Natural Language Processing

Natural Language Processing or NLP is a subject of AI that create the ability to read, understand meaning from our
languages to machines. As you can imagine it's not quite simple but I'm going to show how perform your first steps in this
wide and beautiful field.

Demo - Try it yourself

I have deployed the Chatbot application as a demo here:
http://iris-python-suite.eastus.cloudapp.azure.com:8080

How does it work?

Machine Learning

First is good to know that Machine Learning has a different paradigm compared to common software development.
The main point that is hard to understand is the cycle of development of machine learning models.

Shallow explanation alert

A standard application development cycle are like:

Develop Code->Test (with development data)->Deploy(use real data)

And Machine Learning the Code by itself don't has the same value. It shares the responsability with data! And no any data,
real data! Because the final code to be executed is generated by a merge between development concepts and the data used.
So a machine learning application cycle should be like:

Develop(Train) Model+Real Data->Validate->Deploy the result of this (a Model)

How to train a Model?

There are a lot of techniques to train models and each case and objective needs a big learning curve. In this case I use
the ChatterBot library that encapsulate some techniques and provide train
methods and pre-processed training data to helps us to focus on results.

Pre-trained Model Languages and Custom Model

You can start with this to have a basic conversational chatbot. You also can create all the data to train your chatbot,
that can be perfect to your needs but terrible to make in short time. I this project I use en_core_web_sm as the base of
conversation and merge with custom training data that you can create by a form

Basic Architecture

What did I use in Python

In this application environment I use Python 3.7 with these modules:

PyYAML<=5.0.0
dash==1.12.0
dash-bootstrap-components==0.10.1
dash-core-components==1.10.0
dash-html-components==1.0.3
dash-renderer==1.4.1
dash-table==4.7.0
plotly==4.7.1
numpy==1.18.4
networkx==2.4
Flask>=1.0.0
chatterbot>=1.0.0
chatterbot-corpus>=1.2.0
SQLAlchemy>=1.2
./nativeAPI_wheel/irisnative-1.0.0-cp34-abi3-linux_x86_64.whl

Project Structure

This project has a simple structure to be easy to understand. On the main folder we have 3 most important subfolders:

./app: with all the application code and installing configuration.
./iris: with the InterSystems IRIS dockerfile preparing to serve the application.
./data: To link the host to the container environment by a volume

Application Structure

Now inside the ./app directory we can see some files:

chatbot.py : with the implementation of the web application
iris_python_suite.py : a class with some accelerators to use with IRIS Database and Python by the IRIS Native API.

Database Structure

This application uses Intersystems IRIS as a repository, the globals used are:

^chatbot.training.data: stores all custom training data in the format of question and answers.
^chatbot.conversation : stores all conversation payload.
^chatbot.training.isupdated : controls the training pipeline.

Merchandise of my other solution

I didnt create a report to all conversations but it isnt a problem, with my global graph viewer I can follow the
conversations.

Running the application by yourself

Prerequisites

git
docker and docker-compose (and more memory settings in docker at least 4GB)
acess to a terminal in your environment

Steps

With docker-compose you can easily up one environment with all the pieces and configurations go to the iris-python-covid19
folder and type this:

$ docker compose build
$ docker compose up

Estimated time to up containers

1st time running will depend of your internet link to download the images and dependencies.
If it last more than 15 minutes probably something goes wrong feel free to communicate here.
After the 1st time running the next ones will perform better and take less then 2 minutes.

If is everything ok

After a while you can open your browser and go to the address:

The training data form

http://localhost:8050/chatbot-training-data

The chatbot

http://localhost:8080

You should look at IRIS Admin Portal

I'm using for now the USER namespace

http://localhost:9092
user: _SYSTEM
pass: theansweris42

Using Python to Represent Globals in a Network Chart

Renato Banzai — Tue, 25 Aug 2020 01:55:49 +0000

Globals as a Graph Data Structure in Python

How to use the IRIS Native API + Python to see globals as a Graph Network Chart.

Iris Globals

Reading the documentation these topics are related to globals:

A global consists of a set of nodes (in some cases, only one node), identified by subscripts.
Each node can contain a value.
ObjectScript includes functions to iterate through the nodes of a global and quickly access values.
A global is automatically stored in the database. When you assign a value to a node of a global variable, the data is written immediately to the database. - You can see the contents of a global via an ObjectScript command or via the Management Portal.

A Python Way to See Globals

As one of representations of globals can be a Graph Data Structure there are some modules in Python that can transform
these globals in a visualizable graph.

The Chart Application

Demo - Try it yourself

I have deployed the application as a demo here, my IRIS Database has one global to test ^computer:
(http://iris-python-suite.eastus.cloudapp.azure.com/global-chart)

Into the code

Clone my repository to see all the code implementation.

$ git clone https://github.com/renatobanzai/iris-python-covid19.git

What did I use in Python

In this application environment I use Python 3.7 with these modules.

PyYAML==5.3.1
dash==1.12.0
plotly==4.7.1
networkx==2.4
numpy==1.18.4
dash-bootstrap-components==0.10.1
irisnative-1.0.0-cp34-abi3-linux_x86_64.whl

Project Structure

This project has a simple structure to be easy to understand. On the main folder we have 3 most important subfolders:

./app: with all the application code and installing configuration.
./iris: with the InterSystems IRIS dockerfile preparing to serve the application.
./data: with the files from Johns Hopkins University to ingest and a YAML to change configuration outside the container environment by a volume

Application Structure

Now inside the ./app directory we can see some files:

__main__.py : with the implementation of the web application
iris_python_suite.py : a class performing all data transformation to convert the globals into a networkx graph.

Database Structure

This application uses Intersystems IRIS as a repository, the globals used are:

-^computer : A global to test the graph. If you want, you can test with all other globals default in the USER Namespace.

There are some other globals created by the application that can be used as a test too:

-^config : with some config data
-^raw.covid19 : where the raw data (Source of Data) are ingested
-^countrydetails : to get the population of each country
-^end.date.deaths : to serve the chart requisitions and here is the goal, Its fast!
-^end.timeless.deaths : to server another kind of chart requisition

App Structure

iris_python_suite.py: Inside this file are 2 classes that makes the job:

irisdomestic: Has the same features of irisnative + creates instances of irisglobalchart, irisglobal, etc (factory pattern)
irisglobalchart: Make a recursive track into the global, converting all the data in a Graph Networkx.

Why do I need to convert the Graph into a Networkx Object?

If are you asking yourself, the module networkx has a function position nodes using Fruchterman-Reingold
force-directed algorithm.

Algorithm Fruch... WHAT?

As a graph can have any shape is too hard to represent it in a generic way. This is on algorithm to represent graphs
without a lot of confusion.

The line that perform the use of this algorithm is on python_suite_global.py:

    def get_fig(self):
    _nx = self.obj_nx
    pos = nx.spring_layout(_nx)