How to add AI image generation to your application

This is what you will be building - a Twitter chatbot that converts text prompt into an image. And at the end of this tutorial you’ll be able to do AI image generation for your own application and use case. So let’s get started!‍

A simple example is Tui Casso, a Twitter-based artist (half Twitter half Picasso) that will respond you with some amazing art if you tag @tuicasso with a message describing what you want in a painting.

Make your own Tui Casso

For this solution we will use MindsDB (an emerging open-source AI Logic Server) and DALL-E 2 (an AI image generation model from well-known OpenAI).

MindsDB will handle monitoring of the Twitter data layer, communicating with DALL-E and responding back. All this workflow will be automated via MindsDB’s JOBs.

The advantage of using MindsDB is that it allows you to work with multiple AI frameworks and large language models, all in one place. So, you can combine DALL-E with other AI engines, directly get their outputs into a database and / or push it to the application APIs. We plan to cover these examples in the future tutorials, so stay tuned!

What is MindsDB

MindsDB is a service to deploy and manage Artificial Intelligence Logic, enabling developers to ship AI-powered projects from prototyping & experimentation to production in a fast & scalable way.

MindsDB abstracts Generative AI, LLM’s and other AI model output as a virtual tables (AI-Tables) on top of enterprise databases. This increases accessibility within organizations and enables development teams to use their existing skills to build applications powered by AI.

By taking a data-centric approach to AI, MindsDB brings the process closer to the source of the data minimizing the need to build and maintain data pipelines and ETL-ing, speeding up the time to deployment and reducing complexity.

What is Dall-E

DALL-E and DALL-E 2 are deep learning models developed by OpenAI to generate realistic digital images from a description in natural language, called "prompt".

DALL-E was first revealed by OpenAI in January 2021, and uses GPT modified to generate images.

Step-1: Set-up MindsDB

To get started with MindsDB check out the docs and install MindsDB locally using pip or Docker.

‍We are going to use SQL syntax to manage MindsDB (because it abstracts AI models as a database). Feel free to use your preferred SQL editor (for example, DBeaver) if you want.

Step-2: Connect to Dall-E

First, create a MindsDB model that will connect to the OpenAI’s DALL-E engine for generating images (the output is img_url) based on the description (the input is prompt_template). The variable {{text}} is needed for inserting the content of the users’ tweets (we’ll come to it in the next steps).

And let’s give it a little bit of personality by putting more instructions in to DALL-E in the prompt template!

CREATE MODEL mindsdb.dalle
PREDICT img_url
USING
   engine = 'openai',
   mode = 'image',
   prompt_template = '{{text}}, 8K | highly detailed realistic 3d oil painting style cyberpunk by MAD DOG JONES combined with Van Gogh  |  cinematic lighting | happy colors';

Test it out with the following command providing the {{text}} variable in a where statement:

SELECT * 
FROM mindsdb.dalle 
WHERE text = 'a cute robot helping a little kid build a better world';

STEP-3: Connect to Twitter

Now, we need to prepare MindsDB to be able to read and write responses back into Twitter. Before doing this, you will need to sign up for a Twitter dev account. You can follow our docs that explain the sign up process in detail.

‍
Twitter may take a day or so to approve your new dev account. Once you are approved, here are the steps to link your Twitter account to MindsDB (and the video tutorial for your convenience):

‍

• Open developer portal,
• Select the [+ Add app] button to create a new app
• Select [Create new]
• Select “Production” and give it a name
• Copy and populate in the query
• API Key (aka. consumer_key)
• API Key Secret (aka. consumer_secret)
• Bearer Token
• Click Setup on User authentication settings
• On Permissions select: Read and Write
• On Type of App select: Web App, Automated App or Bot
• On App Info: Provide any url for the callback url and website url
• Click Save
• Once you are back in the app settings, click Keys and Tokens
• Generate Access Token and Secret, and populate on the query
• Access Token
• Access Token Secret

We recommend you use the Elevated access allowing you to pull 2m tweets and to avoid "parameters or authentication issue" error you might get sometimes. You can check this step-by-step guide describing how to apply for the Elevated access.

After you get all these sorted, proceed to connect Twitter to MindsDB. Use the same command as if you are connecting to a database:

CREATE DATABASE my_twitter
WITH
  ENGINE = ‘twitter’,
  PARAMETERS = {
   "consumer_key": "your twitter App API key",
   "consumer_secret": "your twitter App API key secret",
   "bearer_token": "your twitter App bearer TOKEN",
   "access_token": "your twitter App Access Token",
   "access_token_secret": "your twitter App Access Token Secret"
  };

This command creates a MindsDB data integration called my_twitter. It behaves like a database and represents Twitter data in a table called tweets that we can use to search for tweets, as well as to write them.

You can use the Twitter API to get a list of tweets with a particular text or hashtag — in the case below ‘tuicasso’:

SELECT id, created_at, author_username, text
FROM my_twitter.tweets
WHERE query = '(@tuicasso OR #tuicasso) -is:retweet' 
AND created_at > '2023-03-20'
LIMIT 20;

Note that the parameter query supports anything that the Twitter API supports as ‘query.’

To write the Tweet, use the INSERT INTO command. Let's test this by tweeting a few things into the MindsDB Twitter account:

INSERT INTO my_twitter.tweets (in_reply_to_tweet_id, text)
VALUES
 (1633439839491092482, 'MindsDB is great! now its super simple to build ML powered apps using JOBS https://docs.mindsdb.com/sql/tutorials/twitter-chatbot'),
 (1634126825377996800, 'Holy!! MindsDB is such a useful tool for developers doing ML https://docs.mindsdb.com/sql/tutorials/twitter-chatbot');

Works like magic, right? Those tweets should now be live on Twitter. You can check your tweet responses here and here.

Note: you can insert any of the values of the tweepy function create_tweet.

STEP-4: Connect to a Database

MindsDB processes your data but doesn’t store it. Therefore, this solution would work more effectively if you have a database where MindsDB can insert the results of the queries. This could be very useful for several reasons:

• Log all tweets (for troubleshooting and analysis)
• Ensure JOBs (described further) run smoothly

If you have a database already created on your own, please follow these docs to connect it to MindsDB. MindsDB supports almost every database on the market. The command looks like this:

CREATE DATABASE give_it_a_name
[WITH] [ENGINE [=] engine_name] [,]
[PARAMETERS [=] {
  "key": "value",
  …
}];

If you don’t have a database, you may create a free database on AWS using this guide.

In the case of Tuicasso, it already has its own Postgres database, and you’ll see it in the following code examples.

STEP-5: Create and test the workflow

First, create a table, where you will backup all the new tweets that the job will be reading:

CREATE OR REPLACE TABLE postgres.recent_tweets (
   SELECT * 
   FROM my_twitter.tweets 
   WHERE query = '(@tuicasso) -is:retweet'
);

Check it:

SELECT DISTINCT * 
FROM postgres.recent_tweets;

Next, create a table that contains the tweet ids that you already have responded to. Since you want to only respond to new messages, you can populate it with the recent_tweets

CREATE OR REPLACE TABLE postgres.responded_to  ( 
    SELECT DISTINCT id 
    FROM postgres.recent_tweets 
);

Check it:

SELECT * 
FROM postgres.responded_to;

Truncate the responsed_to table while you prepare the rest of the program. This is the equivalent of “you have not responded to anyone yet”.

SELECT * FROM postgres ( 
    TRUNCATE TABLE responded_to 
);

Create a view that contains only deduped tweets:

CREATE VIEW mindsdb.tuicasso_waiting_for_response (
   SELECT * FROM postgres (
       SELECT DISTINCT * 
       FROM recent_tweets 
       WHERE (id not in (SELECT DISTINCT id FROM responded_to)) 
       AND (id is not null)
   )
);

Check it:

SELECT * 
FROM mindsdb.tuicasso_waiting_for_response;

Now, test the query that takes all the tweets waiting for response and creates an image for it using JOIN with the DALL-E model created at the STEP-2. And make it a view:

CREATE VIEW mindsdb.pending_responses_with_images (
   SELECT
       R.id,
       r.text as input_text,
       m.img_url
   FROM mindsdb.tuicasso_waiting_for_response r
   JOIN mindsdb.dalle m
);

This may take a while since, it needs to generate an image for every tweet. Check the status:

SELECT * 
FROM mindsdb.pending_responses_with_images;

Create a table to store the final tweet responses that will contain some fixed text and a link to generated image:

CREATE OR REPLACE TABLE postgres.responses_with_media (
   SELECT
       id as in_reply_to_tweet_id,
       'Voila!' as text,
       Input_text,
       img_url as media_url
   FROM mindsdb (SELECT * FROM mindsdb.pending_responses_with_images)
);

Check how it went:

SELECT * 
FROM postgres.responses_with_media;

Truncate it, as the JOB you are about to create will populate this shortly:

SELECT * FROM postgres (
    TRUNCATE TABLE responses_with_media
);

STEP-6: Automate the workflow

Now you will automate all the workflow with MindsDB JOBs. The idea is simple — you give it the query you want to execute and define how often.

Create a JOB called tuicasso_chatbot_job, which runs every minute and does the following:

• Checks for new tweets and saves them into the tweets table
• Generates images for the tweets it has not replied yet using DALL-E
• Publishes tweets with the content
• Logs the responded tweets

CREATE JOB tuicasso_chatbot_job AS (
   -- update new tweets into the tweets table
   INSERT INTO postgres.recent_tweets (
       SELECT * FROM my_twitter.tweets
       WHERE
           query = '(@tuicasso) -is:retweet'
           AND created_at > "{{PREVIOUS_START_DATETIME}}"
   );
   -- insert media that needs responses, that hasn't been responded to
   INSERT INTO postgres.responses_with_media (
       SELECT
           id as in_reply_to_tweet_id,
           'Voila!' as text,
           Input_text,
           img_url as media_url    
       FROM mindsdb (SELECT * FROM mindsdb.pending_responses_with_images)
   );
   -- insert tweets the responses with media
   INSERT INTO my_twitter.tweets (in_reply_to_tweet_id, text, media_url)
       SELECT in_reply_to_tweet_id, text, media_url 
       FROM postgres.responses_with_media
       WHERE (in_reply_to_tweet_id is not null) 
       AND (in_reply_to_tweet_id not in 
               (SELECT DISTINCT id FROM postgres.responded_to));
   -- log the ones that have been responded to
   INSERT INTO postgres.responded_to (
       SELECT DISTINCT in_reply_to_tweet_id as id 
       FROM postgres.responses_with_media
       WHERE (in_reply_to_tweet_id is not null) 
       AND (in_reply_to_tweet_id not in 
               (SELECT DISTINCT id FROM postgres.responded_to))
   )
)
START '2023-05-09 11:21:00'
EVERY minute;

Finally, explore the job history:

SELECT * 
FROM jobs_history 
WHERE name = 'tuicasso_chatbot_job';

Conclusions

Now you know how to build a cool Twitter chatbot like @tuicasso and if you get an idea of how MindsDB works, you can reproduce similar solutions for your own use case.

MindsDB supports multiple AI frameworks and has hundreds of data integrations. The possibilities are almost endless, so feel free to browse other tutorials and if you need advice or help, just ask in the Community Slack.

It is a large responsive open-source community of AI professionals that constantly builds new tutorials and integrations, and is eager to help you!

‍Good luck with your AI development!