DEV Community: Vinicius Mesel

Wordpress needs an URGENT alternative!

Vinicius Mesel — Wed, 09 Oct 2024 12:28:29 +0000

Since 2012, when I started coding and learning about computer science through the online version of CS50, I became aware of the possibility of building websites using a magical CMS called Wordpress.

It was indeed very magic and still is during these days.

It enabled me to get a template, edit it a little bit and grasp the inner possibilities of what a CMS could do to improve my blogging capabilities (almost none).

Recently, Automattic engaged on a legal battle against WP Engine (and Silver Lake, the Private Equity firm that has the majority stake in WP Engine) for a licensing deal covering the usage of Wordpress resources inside WP Engine. (Read the article: Why WordPress founder Matt Mullenweg has gone ‘nuclear’ against tech investing giant Silver Lake)

There is only a minor GIGANTIC issue in this battle, Wordpress is released under GNU Public License v2 (aka GPLv2 for the intimates) which means that the code must be distributed and made available to anyone.

Inside the license's text we have the following disclosure:

For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.

We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software.

I still don't know how this battle between giants is going to end, but I do know that we need to be prepared for the worst.

We need better open source alternatives for deploying CMSs in a simple and easy way, with similar resources (given the limits of the development of the software).

The alternatives must be reliable and must give us total access of the code just as Wordpress do under the GPLv2 and also have a community built around it to advocate for it's further development and enrichment.

One of the alternatives I'm currently studying is Plone: a Python-built CMS that has been around since 2000s and is serving multiple websites (including many Brazilian government agencies, as far as I know).

It's amazing how much you can build with Plone and I hope it's community can lead us to places as far as the Wordpress community brought us.

Open WebUI + talkd.ai/Dialog: RAG deployment and Development made easy with awesome UI!

Vinicius Mesel — Wed, 09 Oct 2024 11:55:28 +0000

Hey fellow devs and open-source enthusiasts! 🎉 We've got some awesome news that's going to supercharge the way you build and interact with RAGs.

We're super excited to announce that Open WebUI is our official front-end for RAG development. It's a total match!

For those who don't know what talkd.ai/Dialog is:

talkd.ai/Dialog: the brain of the RAGs

Now, talkd.ai/Dialog is where the magic of conversation happens. Our dialog API makes it a breeze to build chatbots and virtual assistants that actually understand what users are saying and can ensure appropriate context is being passed to the prompt.

Why talkd.ai/Dialog is a Game-Changer:

Smart as Heck: Uses top-notch natural language processing to get what users mean.
Context-Aware: Keeps track of conversation context so it doesn’t sound like a broken record.
Scalable: Can handle a ton of simultaneous chats without breaking a sweat.
Integration Friendly: Plug it into just about anything – WhatsApp, Telegram, Slack, you name it and you can build your own plugin!

OpenWebUI: The Universal Web Interface

OpenWebUI is an extensible, feature-rich, and user-friendly self-hosted WebUI designed to operate entirely offline. It supports various LLM runners, including Ollama and OpenAI-compatible APIs. For more information, be sure to check out our Open WebUI Documentation.

Key Features of OpenWebUI:

Effortless Setup: Install seamlessly using Docker or Kubernetes for a hassle-free experience
Ollama/OpenAI API Integration: Effortlessly integrate OpenAI-compatible APIs for versatile conversations alongside Ollama models. Customize the OpenAI API URL to link with LMStudio, GroqCloud, Mistral, OpenRouter, and more.
Pipelines, Open WebUI Plugin Support: Seamlessly integrate custom logic and Python libraries into OpenWebUI using Pipelines Plugin Framework. Examples include Function Calling, User Rate Limiting, Usage Monitoring, Live Translation, Toxic Message Filtering, and much more.

What do I need to do to use talkd.ai/dialog with Open WebUI?

To use both, you just need to clone talkd.ai/dialog repository from GitHub using the command below:

git clone https://github.com/talkdai/dialog.git

And run

docker-compose -f docker-compose-open-webui.yml

This will build the docker and will enable you to access your localhost:3000 server of Open WebUI with talkd.ai already setup.

If everything went fine, you will be able to sign up locally on Open WebUI and reach a page like the one below:

Hope you like this collab!

Hugs from me and the talkd.ai team!

Open WebUI + talkd.ai/Dialog: RAG deployment and Development made easy with awesome UI!

Vinicius Mesel — Tue, 25 Jun 2024 20:19:05 +0000

Hey fellow devs and open-source enthusiasts! 🎉 We've got some awesome news that's going to supercharge the way you build and interact with RAGs.

We're super excited to announce that Open WebUI is our official front-end for RAG development. It's a total match!

For those who don't know what talkd.ai/Dialog is:

talkd.ai/Dialog: the brain of the RAGs

Why talkd.ai/Dialog is a Game-Changer:

Smart as Heck: Uses top-notch natural language processing to get what users mean.
Context-Aware: Keeps track of conversation context so it doesn’t sound like a broken record.
Scalable: Can handle a ton of simultaneous chats without breaking a sweat.
Integration Friendly: Plug it into just about anything – WhatsApp, Telegram, Slack, you name it and you can build your own plugin!

OpenWebUI: The Universal Web Interface

Key Features of OpenWebUI:

Effortless Setup: Install seamlessly using Docker or Kubernetes for a hassle-free experience
Ollama/OpenAI API Integration: Effortlessly integrate OpenAI-compatible APIs for versatile conversations alongside Ollama models. Customize the OpenAI API URL to link with LMStudio, GroqCloud, Mistral, OpenRouter, and more.
Pipelines, Open WebUI Plugin Support: Seamlessly integrate custom logic and Python libraries into OpenWebUI using Pipelines Plugin Framework. Examples include Function Calling, User Rate Limiting, Usage Monitoring, Live Translation, Toxic Message Filtering, and much more.

What do I need to do to use talkd.ai/dialog with Open WebUI?

To use both, you just need to clone talkd.ai/dialog repository from GitHub using the command below:



git clone https://github.com/talkdai/dialog.git

And run



docker-compose -f docker-compose-open-webui.yml

This will build the docker and will enable you to access your localhost:3000 server of Open WebUI with talkd.ai already setup.

If everything went fine, you will be able to sign up locally on Open WebUI and reach a page like the one below:

Hope you like this collab!

Hugs from me and the talkd.ai team!

talkd.ai got accepted into the Github Accelerator! (also our first official release)

Vinicius Mesel — Thu, 23 May 2024 16:00:00 +0000

Hey there!

If you still don’t know us, we are talkd.ai, an open-source organization that is maintaining Dialog, a project focused on letting you easily deploy any LLM that you want (currently any of those available in the Langchain and partner libraries - we will cover more on that later).

Today, we are very grateful for announcing two things:

This is our first official public release, our "Numero Uno" and a starting point to this adventure that will be long and fun.
Now it's official: We got accepted in the GitHub Accelerator 2024 AI Cohort - a Cohort full of amazing people that started on April 22nd.

What is the GitHub Accelerator?

The GitHub accelerator is an accelerator that is running its second cohort in a slightly different approach from the usual: it focuses on making open-source sustainable and helping maintainers find sustainable ways to fund their work full-time on the projects.

During the accelerator, you will be connected to references from fields such as AI, InfoSec, Successful Open Source maintainers, investment funds, and many other professionals who will guide you through the many possibilities of open-source funding.

The project also provides you a stipend for 10 weeks, allowing you and your team to be full-time focused on the development of software and communications of your project, credits on OpenAI and Microsoft Azure.

Back to the project's history: How did the project start?

The project started to help us (Thiago Avelino and I) create chat experiences that resembled human behavior in answering frequently asked questions inside our contexts (Avelino inside Buser and my context of wanting to learn more about LLM deployments and maintenance).

Still, as the project grew, our contexts changed a lot, as the necessity for different techniques, retrievers, optimizations, and plugins

Nowadays, the project allows you to deploy any model that respects Langchain LCELs or our libraries' chain model, you can choose which one to maintain and set up from there.

The process of getting to where we are right now involves lots of people, but I would like to specially:

Luan Fernandes - our Langchain specialist and long-time contributor
Walison Filipe - our FastAPI master and testing guru
Gregg and Kevin from the GitHub Accelerator - on helping us improve our software through communications, mentorship, invaluable resources and connections
Andreas, Alicia, Namee, and Jurgen - you have amazing projects and invaluable lessons on improving pitching.

What can I do with talkd.ai/dialog?

talkd.ai/dialog lets you deploy any LLM that you want (with the code already adapted to the LCEL or AbstractLLM models) in 5 minutes.

With a simple docker-compose up you have sample data and a sample prompt up and running.

If you want to customize, you can use our library: dialog-lib, to implement custom RAGs and Retrievers. We are fully integrated with SQLAlchemy, PGVector, Anthropic, and OpenAI.

Here is our quick product demo to showcase you how simple is to deploy our software:

Here is the official release link from GitHub.

GPT-4o: Learn how to Implement a RAG on the new model, step-by-step!

Vinicius Mesel — Tue, 14 May 2024 02:48:00 +0000

OpenAI just released their latest GPT model, GPT-4o, which is half of the price of the current GPT-4 model and way faster.

In this article, we will show you:

What is a RAG
Getting started: Setting up your OpenAI Account and API Key
Using Langchain's Approach on talkdai/dialog
Using GPT-4o in your content

PS: Before following this tutorial, we expect you to have a simple knowledge of file editing and docker.

What is a RAG?

RAG is a framework that allows developers to use retrieval approaches and generative Artificial Intelligence altogether, combining the power of a large language model (LLM) with an enormous amount of information and a specific custom knowledge base.

Getting started: Setting up your OpenAI Account and API Key

To get started, you will need to access the OpenAI Platform website and start a new account, if you still don't have one. The link to get into the form directly is: https://platform.openai.com/signup

Whenever you submit the form with your e-mail and password, you will receive an e-mail (just like the one below) from OpenAI to activate and verify your account.

If you, for some reason, don't see this e-mail in your inbox, try checking your spam. If it's still missing, just hit the resend button on the page to verify your e-mail address.

And... Congrats! You now are logged in to OpenAI Platform.

Now, let's generate your API Token to be able to use the Chat Completion endpoint for GPT-4o, allowing you to use it with talkdai/dialog, a wrapper for Langchain's with a simple-to-use API.

On the left menu, you will be able to find a sub-menu item called: API Keys, just click on it and you will be redirected to the API Keys.

If it's your first time setting up your account, you will be required to verify your phone number.

After you verify your phone number, the Create new secret key will be enabled as the following image shows:

OpenAI keys are now project-based, which means you will be assigned a project as a "host" of this generated key on your account, it will enable you to have more budget control and gain more observability on costs and tokens usage.

To make the process easier, we will just name our API Key and use the "All" permission type, for production scenarios, this is not the best practice and we recommend you set up permissions wisely.

After you hit Create, it will generate a unique secret key that you must store in a safe place, this key is only visualizable once, so if you ever need to get it again, you will need to get it from the place you saved or generate a new one.

Using Langchain's approach on talkdai/dialog

What is Langchain?

Langchain is a framework that allows users to work with LLM models using chains (a concept that is the sum of a prompt, an LLM model, and other features that are extensible depending on the use case).

This framework has native support for OpenAI and other LLM models, granting access to developers around the globe to create awesome applications using a Generative AI approach.

What is talkdai/dialog?

talkdai/dialog, or simply Dialog, is an application that we've built to help users deploy easily any LLM agent that they would like to use (an agent is simply an instance of a chain in the case of Langchain, or the joint usage of prompts and models).

The key objective of talkdai/dialog is to enable developers to deploy LLMs in less than a day, without having any DevOps knowledge.

Setting up Dialog

On your terminal, in the folder of your choice, clone our repository with the following command, in it you will have the basic structure to easily get GPT-4o up and running for you.



git clone https://github.com/talkdai/dialog

In the repository, you will need to add 3 files:

Your .env file, where that OpenAI API Key will stay stored (please, don't commit this file in your environment). This will solely based on the .env.sample file we have on the root of the repository.
The file where you will define your prompt (we call it prompt.toml in the docs, but you can call it whatever you want)
And, last but not least important, a CSV file with your content.

.env file

Copy and paste the .env.sample file to the root directory of the repository and modify it using your data.



PORT=8000 # We recommend you use this as the default port locally
OPENAI_API_KEY= # That API Key we just fetched should be put here
DIALOG_DATA_PATH=./know.csv # The relative path for the csv file inside the root directory
PROJECT_CONFIG=./prompt.toml # The relative path for your prompt setup
DATABASE_URL=postgresql://talkdai:talkdai@db:5432/talkdai # Replace the existing value with this line
STATIC_FILE_LOCATION=static # This should be left as static
LLM_CLASS=dialog.llm.agents.lcel.runnable # This is a setting that define's the Dialog Model Instance we are running, in this case we are running on the latest LCEL version

Prompt.toml - your prompt and model's settings

This file will let you define



[model]
model_name = "gpt-4o"
temperature = 0.1

[prompt]
prompt = """
You are a nice bot, say something nice to the user and try to help him with his question, but also say to the user that you don't know totally about the content he asked for.
"""

This file is quite straightforward, it has 2 sections: one for defining model details, which allows us to tweak temperature and model values.

The second section is the most interesting: it's where you will define the initial prompt of your agent. This initial prompt will guide the operation of your agent during the instance's life.

Most of the tweaks will be done by changing the prompt's text and the model's temperature.

Knowledge base

If you want to add more specific knowledge in your LLM that is very specific to your context, this CSV will allow you to do that.

Right now, the CSV must have the following format:

category - the category of that knowledge;
subcategory - the subcategory of that knowledge;
question - the title or the question that the content from that line answers, and
content - the content in itself. This will be injected in the prompt whenever there is a similarity in the user's input with the embeddings generated by this content.

Here is a sample example:



category,subcategory,question,content
faq,football,"Whats your favorite soccer team","My favorite soccer team is Palmeiras, from Brazil. It loses some games, but its a nice soccer team"

Langchain's behind the scene

In this tutorial, we will be using an instance of a Dialog Agent based on Langchain's LCEL. The code below is available in the repository you just cloned.



# For the sake of simplicity, I've removed some imports and comments

chat_model = ChatOpenAI(
    model_name="gpt-3.5-turbo",
    temperature=0,
    openai_api_key=Settings().OPENAI_API_KEY,
).configurable_fields(
    model_name=ConfigurableField(
        id="model_name",
        name="GPT Model",
        description="The GPT model to use"
    ),
    temperature=ConfigurableField(
        id="temperature",
        name="Temperature",
        description="The temperature to use"
    )
)

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            Settings().PROJECT_CONFIG.get("prompt").get("prompt", "What can I help you with today?")
        ),
        MessagesPlaceholder(variable_name="chat_history"),
        (
            "system",
            "Here is some context for the user request: {context}"
        ),
        ("human", "{input}"),
    ]
)

def get_memory_instance(session_id):
    return generate_memory_instance(
        session_id=session_id,
        dbsession=next(get_session()),
        database_url=Settings().DATABASE_URL
    )

retriever = DialogRetriever(
    session = next(get_session()),
    embedding_llm=EMBEDDINGS_LLM
)

def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

chain = (
    {
        "context": itemgetter("input") | retriever | format_docs,
        "input": RunnablePassthrough(),
        "chat_history": itemgetter("chat_history")
    }
    | prompt
    | chat_model
)

runnable = RunnableWithMessageHistory(
    chain,
    get_memory_instance,
    input_messages_key='input',
    history_messages_key="chat_history"
)

The first lines are where we define our model temperature and name. By default, the model will be gpt-3.5-turbo and the temperature 0, but since we defined it in the prompt configuration file, it will be changed to gpt-4o and the temperature to 0.1.

To send a prompt inside Langchain, you need to use its template, which is what we do next on the ChatPromptTemplate.from_messages instance.

In the next lines, we define the memory, how it should be consumed and the instance of the retriever (where we are going to grab our custom data from).

Using GPT-4o in your content

After this long setup, it's time to run our application and test it out. To do it, just run:



docker-compose up --build

With this command, your docker should be up and running after the image builds.

After the logs state: Application startup complete., go to your browser and put the address you've used to host your API, in the default case, it is: http://localhost:8000.

Go to the /ask endpoint docs, fill in the JSON with the message you want to ask GPT and get the answer for it like the following screenshot:

That's all! I hope you enjoyed this content and using talkdai/dialog as your way-to-go app for deploying Langchain RAGs and Agents.

Deploy your own ChatGPT in 5 minutes

Vinicius Mesel — Thu, 25 Apr 2024 02:21:00 +0000

Disclaimer: The files for this tutorial are available at: our Github Tutorial Repo

If you ever wondered about how you could deploy your own version of ChatGPT with your/your company's content, this is your time to shine: I have a wonderful solution for you.

For those who don't know me or my project, I'm Vinnie, a software engineer and co-founder of the Talkd.AI project. We are a simple and easy to deploy app that allows any developer with some context of Docker and Python to deploy their own RAG (Retrieval-augmented Generation) the name of the type of applications that use a (vector) database and your custom content to return data from and to your LLM.

What you will need to have in order to use our Dialog server?

A computer/server with a docker instance up and running
A CSV file with some contents from your company (we will specify the format later on)
A TOML file for setting up your model and prompts
An OpenAI API Key (here is a tutorial to generate one)

Getting started

In order to start your own deployment of our Dialog server, you will need to use the following docker-compose file:



services:
  db:
    image: pgvector/pgvector:0.6.2-pg15
    restart: always
    ports:
      - '5432:5432'
    environment:
      POSTGRES_USER: talkdai
      POSTGRES_PASSWORD: talkdai
      POSTGRES_DB: talkdai
    volumes:
       - ./etc/db-extensions.sql:/docker-entrypoint-initdb.d/db-extensions.sql
    healthcheck:
      test: ["CMD", "pg_isready", "-d", "talkdai", "-U", "talkdai"]
      interval: 10s
      timeout: 5s
      retries: 5
  dialog:
    image: ghcr.io/talkdai/dialog:latest
    volumes:
      - ./data/:/app/data/
    ports:
      - '8000:8000'
    depends_on:
      db:
        condition: service_healthy
    environment:
      - PORT=8000
      - DATABASE_URL=postgresql://talkdai:talkdai@db:5432/talkdai
      - OPENAI_API_KEY=sk-your-openai-api-key
      - STATIC_FILE_LOCATION=/app/static
      - DIALOG_DATA_PATH=../data/your.csv
      - PROJECT_CONFIG=../data/your.toml

PGVector - Our first service

The first service of this docker-compose file is a PostgreSQL instance equipped with pgvector, our vector database implementation of choice since it's already inside a major open-source database vendor.

On this service, we've configured a volume mapping to a file, this file's content is the following:



CREATE EXTENSION IF NOT EXISTS vector;

This file only enables the vector plugin if it doesn't exist already in your database. If you are using another implementation of a PGVector Docker image that already runs this command, you can ignore this volume mounting.

Second Service - Dialog Server

As you can see, the second service is our Dialog Server, it's our implementation of the RAG approach built on top of FastAPI, LangChain and PGVector Python extension.

We also have a volume mapping inside this service:



    volumes:
      - ./data/:/app/data/

This volume mapping allow us to provide our prompt information and your custom content in the CSV format.

The CSV file for loading your own ChatGPT must have the following columns:

category - The category of that content
subcategory - The subcategory of that content
question - The question/title of that content
content - The content in itself, this content will generate the embeddings for our database.

On our example, we will add some questions regarding sports:



category,subcategory,question,content
faq,football,"Whats your favorite soccer team","My favorite soccer team is Palmeiras, from Brazil."
faq,football,"Whats your favorite soccer player","My favorite soccer player is Neymar, from Brazil."

Inside our prompt file (the .toml one I've mentioned before), we will configure our prompt and how the bot will speak, which model and temperature it will use and a fallback message in order it can generate answers on topics that it didn't find a content for a certain user message.



[model]
model = "gpt-3.5-turbo-1106"
temperature = 0.1

[prompt]
header = """You are Robert, an AI bot that answers any questions that a user may have."""
question_signalizer = "Answer the following question:"

suggested = """Just answer the following question, don't change context and don't reply using more than 50 words."""

fallback = """
Your a nice bot, say something nice to the user and try to help him with his question, but also say to the user that you don't know totally about the content he asked.
"""

Finishing the setup

With all of these files saved inside the ./data/ directory, we are able to do the last part of the setup: setting our environment variables in order to deploy our bot!

The environment variables that must be filled are:

PORT - the Port where the web server is allowed to be started on (we recommend using 8000, but you can set your own)
DATABASE_URL - the database URL (like postgresql://talkdai:talkdai@db:5432/talkdai)
OPENAI_API_KEY - The generated OpenAI API Key
DIALOG_DATA_PATH - The relative path from /app/ for the data CSV file you mapped inside your docker compose volume. (Our standard is to use the folder /app/data to host this file, so the value of this variable must be: ../data/your.csv
PROJECT_CONFIG - The Prompt file location. The explanation is the same as above, but for the toml file. In this tutorial, we will set this as ../data/your.toml

Final step: Deploying and running

Now that we finally have our setup done, we can start our Dialog Server instance by running:



docker-compose up # -d # if you want to run it on background

You will need to wait a while until it builds everything, get our pgvector instance up and running and create the embeddings for your custom content.

When it finishes, your docker-compose log will look like this:

To access the API Swagger, you will go in your browser and use the following URL (or the equivalent for your setup): http://localhost:8000/docs.

If everything is fine, the following page must have loaded and you can now start asking questions to your own ChatGPT instance.

Using the /ask endpoint, you will be able to send any message you want the ChatGPT to answer, such as Whats your NBA team of heart?:

And after hitting send, the answer comes:

Conclusion

If you follow through this tutorial, you will be able to deploy your own ChatGPT with custom content for your company or your website.

Leave a like in this post if you enjoyed and share it with your friends.

Also, if you want more information on dialog and talkd.ai, access our GitHub page: https://github.com/talkdai/

pRESTd release 1.1.4 - Query performance improvement

Vinicius Mesel — Tue, 08 Nov 2022 18:04:13 +0000

If you want to use your database as a data source for an API you are creating, but you are not happy with building it from scratch: pRESTd is for you!

Basically pRESTd is a software that enables you to perform queries through API calls on your whole database system, improving delivery time for your development team.

In this new release (1.1.4), we were able to work on a performance improvement on our querying system. Our base query used to use the following query:

select json_agg(s) from (select * from your_awesome_table) s

Which, on our analysis, was 5x slower than using JSONB aggregation function.

Benchmark

As we can see above, implementing the functionality in Golang gives us better performance on smaller scenarios, but as we grow our data requests, the linearity of the equation is not followed.

That's why we chose to implement with JSONB_AGG as our serializer on the database end, enabling us to save time on development and test writing.

You can see the requests data available in our PR, on the subtitles below.

With JSON_AGG()

2022/11/04 16:54:35 [warning] adapter is not set. Using the default (postgres)
2022/11/04 16:54:35 [warning] command.go:920 You are running prestd in debug mode.
[prestd] listening on 0.0.0.0:80 and serving on /
2022/11/04 16:54:41 [debug] server.go:2084 generated SQL:SELECT json_agg(s) FROM (SELECT * FROM "database"."public"."table" LIMIT 1000 OFFSET(1 - 1) * 1000) s parameters: []
[negroni] 2022-11-04T16:54:41Z | 200 |   5.11392325s | 54.186.223.54 | GET /database/public/table
2022/11/04 16:55:18 [debug] server.go:2084 generated SQL:SELECT json_agg(s) FROM (SELECT * FROM "database"."public"."table" LIMIT 100 OFFSET(1 - 1) * 100) s parameters: []
[negroni] 2022-11-04T16:55:18Z | 200 |   504.78063ms | 54.186.223.54 | GET /database/public/table

With JSONB_AGG()

[prestd] listening on 0.0.0.0:80 and serving on /
2022/11/04 16:59:26 [debug] server.go:2084 generated SQL:SELECT jsonb_agg(s) FROM (SELECT * FROM "database"."public"."table" LIMIT 100 OFFSET(1 - 1) * 100) s parameters: []
[negroni] 2022-11-04T16:59:26Z | 200 |   479.260256ms | 54.186.223.54 | GET /database/public/table
2022/11/04 17:00:05 [debug] server.go:2084 generated SQL:SELECT jsonb_agg(s) FROM (SELECT * FROM "database"."public"."table" LIMIT 1000 OFFSET(1 - 1) * 1000) s parameters: []
[negroni] 2022-11-04T17:00:05Z | 200 |   1.912713761s | 54.186.223.54 | GET /database/public/table

About the benchmark

We used a table with 56 million rows, two indexes (on the primary date and id fields) and both of the API calls (and queries) were paginated.

What the heck are Python Generator Expressions and List Comprehensions?

Vinicius Mesel — Sun, 10 Feb 2019 12:29:58 +0000

Have you ever thought on initializing a list with its values with a one liner expression? That's totally possible in Python using list comprehensions. But sometimes you wont use the list stored value for a while and you just wanted to initialize the values for a future use... That's a simple use case of or Generator Expressions.

PS: If you have read Fluent Python from @ramalhoorg , there is nothing new right here, but you can share this text to a friend, so she will be able to learn more about Python.

PS 2: Thanks @ramalhoorg for the examples on the book, they were very useful and some of them are used right here!

What are List Comprehensions?

List comprehensions are a list expression that creates a list with values already inside it, take a look at the example below:

>>> my_incredible_list = [x for x in range(5)]
>>> my_incredible_list
[0, 1, 2, 3, 4]

This list comprehension is the same as if you were doing a for loop appending values to a list.

>>> my_incredible_list = []
>>> for x in range(5):
>>>     my_incredible_list.append(x)
>>> my_incredible_list
[0, 1, 2, 3, 4]

With list comprehensions, you are able to generate lists with tuples inside iterating two other lists.

>>> foods = "apple banana sausages".split() # We declare a list here with foods
>>> beverages = "coca-cola pepsi guarana".split() # We declare a list of beverages
>>> [(food, beverage) for food in foods for beverage in beverages]
[('apple', 'coca-cola'), ('apple', 'pepsi'), ('apple', 'guarana'), ('banana', 'coca-cola'), ('banana', 'pepsi'), ('banana', 'guarana'), ('sausages', 'coca-cola'), ('sausages', 'pepsi'), ('sausages', 'guarana')]

What are Generator Expressions?

So generator expressions are just like a list comprehension, but it doesn't store it's values on the memory, it only generate the value you are iterating on when the generator is called. Take a look at the example below:

>>> my_incredible_list = (x for x in range(5))
>>> my_incredible_list
<generator object <genexpr> at 0x7f14149a3db0>

To check for a value on a generator you can either iterate over the generator using a for loop or you can check the values using the next() operator.


>>> # Using the next operator
>>> next(my_incredible_list)
0
>>> next(my_incredible_list)
1
>>> next(my_incredible_list)
2 
>>> next(my_incredible_list)
3
>>> next(my_incredible_list)
4
>>> next(my_incredible_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>
>>> # Using a for loop
>>> for item in my_incredible_list:
...     print(item)
... 
0
1
2
3
4

Checking out the Assembly of both of them to see the differences

Assembly instructions from Python disassambler

>>> import dis
>>> my_incredible_list = [x for x in range(5)]
>>> dis.dis(my_incredible_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/dis.py", line 67, in dis
    type(x).__name__)
TypeError: don't know how to disassemble list objects
>>> my_incredible_list = (x for x in range(5))
>>> dis.dis(my_incredible_list)
  1           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                10 (to 14)
              4 STORE_FAST               1 (x)
              6 LOAD_FAST                1 (x)
              8 YIELD_VALUE
             10 POP_TOP
             12 JUMP_ABSOLUTE            2
        >>   14 LOAD_CONST               0 (None)
             16 RETURN_VALUE
>>>

So as we see right here, the disassemble of the list comprehension is not possible, because we cannot disassemble a data structure, but we can still disassemble the generator expression.

When we run dis.dis(<genexpr> right here) we are able to see that the expression is treated like a queue, we yield the value and then we pop it, so it's not called anymore.

If we call the dis.dis(<listcomp> here) we are able to see the running procedure to create a list using the listcomp expression:

>>> dis.dis("[x for x in range(5)]")
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x7f14149c84b0, file "<dis>", line 1>)
              2 LOAD_CONST               1 ('<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_NAME                0 (range)
              8 LOAD_CONST               2 (5)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

So as we see, the MAKE_FUNCTION piece is run generating the values for the list, we instantiate the function range, load a value to it (5), we call it and them we save it to the list instead of just yielding the value every time the generator is called.

Nomad Couch: Find a tech nomad to share a place with you

Vinicius Mesel — Sun, 04 Nov 2018 21:20:25 +0000

Hey Guys, My friend Guilherme is going to live in Thai for a year and he is searching for indie hacker/nomad interested in sharing a place with him for a week.

To help Digital Nomads find their couches (like a couch surfing app), we've created Nomad Couch, a online spreadsheet to find your next nomad family to stay with and enjoy new moments together.

Help us with your feedback - http://nomadcouch.xyz

Why Should You Care About Software Maintainability?

Vinicius Mesel — Mon, 22 Oct 2018 02:08:44 +0000

Real quick: This essay is mainly written for those who are not a software developer yet, or don't code as their principal source of income.

Hey guys, as you all may know (for those who don't, I'm going to explain it real quick here) I changed jobs, and now, I'm working for a bank in Brazil (I won't share it's name here as I'm not paid for advertising them). Coding is a side thing in my career there, I mainly work with investment analysis inside Excel (VBA) and other Microsoft applications. ~~Investment things are very nice, but Excel sucks.~~

As I've seen there, lots of people from Investment Banking are using programming languages to get their work done faster and better than before, which is great, but the point is: Are those codes produced by non-software engineer people maintainable in a production scenario?

Software engineers who care about their code quality, make their code thinking of: error handling, data IO, test-driven development, code readability, code complexity and many other aspects that beginners and non-expert people.

Why should I f*****g care about software maintainability?

If your code will only be used by you for a task, you shouldn't care a lot for these, your code needs to solve your problem mainly, but when you are working with multiple people in the same project/field/area/department/whatever, you won't be the only one who will use what you are coding.

For maintainability purposes, Python, created two PEPs that should be followed as a good practice for software engineering. Those PEPs are:

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

As shown above, the simplicity and readability counts when developing new software that will be used.

Imagine if you get into a company and there is no documentation about that code, no deploy instructions, no development setup, just the code there. Would you be able to pick it up and continue working where the other people left it? Probably not, we all need best practices for better code maintenance and developers on-boarding on new systems! Maintaining a big code base without good practices is like this meme:

The best coding practices are those who all developers in your team know and apply, as well as can be passed to new developers. Here are some use cases of best practices that may be useful:

If well written comments (or docstrings) are helpful inside your code base and guide new developers to the maintenance, keep them and update those that need to be updated. Don't keep dull comments, such as:


# Example of a dull comment

a = 1 + 2 # this line adds 1 with 2 and saves to

# Example of a needed comment

def really_cool_function_that_sums_to_numbers(x, y):
    # This function sums X and Y and returns the result

If you need to comment multiple lines of your code to make it understandable, its a good time to think on how to improve it. Code should be readable and easily understandable (PEP 20 lesson here)

Functions should do just a single function (sounds redundant, right?) as like Pure Functions (concept from functional programming that states functions that cannot cause side effects to global variables and other functions).

There are plenty of best practices and code styling guides to follow according to your language of choice. The main point of this text is: remember that other developers will maintain your code on a near future, and you will probably maintain it too and won't remember lots of things. Make your code simple so people understand it and continue your work.

Originally published on: Vmesel's blog