DEV Community: Lorenzo Tenti

The Future of AI and other stories

Lorenzo Tenti — Mon, 19 Feb 2024 18:38:39 +0000

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

Read this 📚

DevTools for language models — predicting the future

This post covers the state of LLM DevTools today and where the authors think it’s heading in the future, touching on topics such as experimentation, prompt design, vector databases, and more.

Watch this 👀

OpenAI CEO on GPT-4, ChatGPT, and the future of AI

In this podcast episode, Lex Fridman interviews Sam Altman, CEO of OpenAI. They cover a wide range of topics, from GPT-4 to the meaning of life.

Hack this 🛠️

Add generative artificial intelligence capabilities to Pandas with PandasAI

PandasAI is a Python library that adds generative artificial intelligence capabilities to Pandas, the popular data analysis and manipulation tool. It makes Pandas conversational, allowing you to ask questions about your data and get answers back, in the form of Pandas DataFrames. It can also create plots from a DataFrame.

pandas_ai.run(
    df,
    "Plot the histogram of countries showing for each the gpd, using different colors for each bar")

Meme this 🚀

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

Build data apps with markdown and SQL & other stories

Lorenzo Tenti — Mon, 19 Feb 2024 18:25:06 +0000

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

The cover picture was generated using ChatGPT 4:

Here's a cartoon representation of SQL functionalities, set in a whimsical data world. This playful illustration brings to life the various SQL operations with colorful, anthropomorphic characters each performing their unique tasks in a fantastical digital realm.

Read this 📚

Open Source Data Engineering Landscape 2024

This post from Practical Data Engineering presents a full exploration of the open-source data engineering landscape for 2024. As usual, it’s both overwhelming and inspiring. I’m sure you’ll discover something new while reading this.

Watch this 👀

The Future of Knowledge Graphs in a World of LLMs

This video discusses the future of knowledge graphs in the context of large language models (LLMs), showing how they will complement each other, allowing us to reap the benefits of both technologies.

Hack this 🛠️

Build polished data products with SQL

Evidence is a lightweight framework for building data apps using markdown and SQL. It's open source and free to get started.

Meme this 🚀

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

Big Data is dead & other stories

Lorenzo Tenti — Tue, 13 Feb 2024 09:57:35 +0000

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

Read this 📚

Big Data is Dead

In this article, Jordan Tigani challenges the idea that big data is necessary for analytics and argues that it’s mainly a result of poor data management practices. The article explains that big data solutions are often overkill for most analytics workloads and that they incur high costs and complexity. In addition, he suggests that modern hardware advancements have made it possible to process large amounts of data on a single machine without the need for distributed computing.

Learn this 🧑🏻‍🏫

Prompt Engineering Guide

Prompt engineering is a hot topic these days, thanks to the rise of large language models (did you catch that?). This guide has everything you need to get started: recent papers, tutorials, videos, links and tools.

Watch this 👀

Advanced Design Patterns for DynamoDB

The first time I watched this talk, I found it mind-blowing. Rick Houlihan reveals the common misconceptions about NoSQL database services. I used to think NoSQL was great for its flexibility, but the opposite is often true. Even if he focuses on DynamoDB, the concepts explained in the talk (single-table design and using composite keys to support different access patterns) can be applied to many other services. Definitely a must-watch.

Hack this 🛠️

There's an AI for that

This website, by Andrei, is a curated list of products offering AI-based solutions to any kind of problem.

Disclaimer: this is not an invite to literally hack the website but only to have fun with one of the tools you’ll find there.

Meme this 🚀

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

SQL should be your default choice & other stories

Lorenzo Tenti — Thu, 08 Feb 2024 10:50:14 +0000

This article originally appeared on ingest this!, a curated newsletter about Data Engineering, MLOps, and Machine Learning Engineering.

Read this 📚

SQL should be your default choice for data engineering pipelines

We all love a bit of SQL, but is it powerful enough to be the only language used in your data engineering pipelines? In this article, Robin Linacre argues that you should look no further than SQL, thanks to modern techniques, such as CTEs, and new tools, such as DuckDB, SQLGlot, and dbt.

Watch this 👀

Andrew Ng on Data-Centric AI

Andrew Ng discusses how and why we should move from big data to good data when building AI systems.

Hear this 🎧

Airflow Sucks for MLOps

Demetrios Brinkmann from MLOps.community interviews Stephen Bailey about working with data platforms, the problems related to the orchestration layer and the limits of the current tools.

Also available as a video here.

Hack this 🛠️

Draw plots on the terminal with YouPlot

YouPlot is a great tool written in Ruby that allows you to create plots directly on your terminal. Many different plot types are available.

Meme this 🚀

How to build a Serverless Twitter bot

Lorenzo Tenti — Wed, 30 Jan 2019 16:22:50 +0000

In this post I'm going to show you how to build a Twitter bot using the Serverless Framework. This framework represents "the easy and open way to build serverless applications", and if you don't know what I'm talking about you should definitely check it out.

We're going to build a bot that will retweet at regular intervals the most relevant tweets about serverless computing, and will follow the tweet's author. Have a look here if you're not afraid of spoilers.

For this tutorial we're using AWS Lambda with the Python 3.6 runtime, but you could easily switch to another provider/programming language.

We are going to interact with the Twitter API using Twython.

Setup the Twitter Account and App

First of all, you need to create a Twitter account for the bot and then go to developer.twitter.com to apply for a developer account to access the Twitter API.

After that, you can create a Twitter App and get the keys and tokens needed. The Twitter interface is easy to use and the process is quite straightforward.

Install Serverless and configure the environment

The easiest way to install the Serverless framework is to follow their installation guide, here, and then setup the AWS credentials as shown here.

When the the framework is installed and configured, a new service can be easily created with:

$ serverless create \
  --template aws-python3 \
  --name serverlessbot \
  --path serverlessbot

Then, cd into the newly created directory and create a Python virtual environment using virtualenv:

$ cd serverlessbot
$ virtualenv venv --python=python3

You will need to activate the virtual environment with:

$ source venv/bin/activate

Now you can install the twython library and then save the package versions of the environment to a requirements.txt file.

$ pip install twython
$ pip freeze > requirements.txt

At this point we should be ready to start coding the bot. However, manually configuring Lambda to use external libraries could be tricky, therefore we're going to use the serverless-python-requirements plugin to make it easier.

First of all, initialize npm with:

$ npm init

accept all the defaults (press enter several times)
then install the plugin with:

$ npm install --save serverless-python-requirements

Finally, add the following at the end of your serverless.yml file:

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

You can find more information on this plugin here.

Programming the bot

Now we can finally start building the bot.

We have to change the functions section in the serverless.yml file to look like:

functions:
  retweet:
    handler: handler.retweet
    events:
      - schedule: rate(2 hours)

This will call the method retweet of handler.py every 2 hours.

Then, we can replace the handler.py file with the following content:

import json, config
from twython import Twython, TwythonError

def retweet(event, context):

    twitter = Twython(config.APP_KEY, config.APP_SECRET, config.OAUTH_TOKEN, config.OAUTH_TOKEN_SECRET)

    search_results = twitter.search(q='serverless', result_type='mixed', lang='en')
    message = ""

    for tweet in search_results['statuses']:
        try:
            twitter.retweet(id=tweet['id'])
            message = f"Retweeted \"{tweet['text']}\" by {tweet['user']['name']}"
            twitter.create_friendship(id=tweet['user']['id'])
            break
        except TwythonError:
            pass

    body = {
        "message": message,
        "input": event
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response

The code is quite self-explanatory. Basically, we are searching for English tweets mentioning serverless, retweeting them, and follow the tweet's author.

The return response is only for logging and testing.

As you can see, the Twitter keys and tokens are read using config. All you need to do is to create a config.py file in the same directory and fill it with the information you got from the Twitter App page:

APP_KEY="xxxyyyzz"
APP_SECRET="xxxyyyzz"
OAUTH_TOKEN="xxxyyyzz-xxxyyyzz"
OAUTH_TOKEN_SECRET="xxxyyyzz"

Deploying the bot

At this point, we're going to deploy the function with:

$ serverless deploy -v

The Serverless Framework will now take care of everything needed to deploy your Lambda function. When the process is finished, you will be able to see it in the Lambda Management Console on AWS. If you notice, there will be a CloudWatch event configured to fire the function every 2 hours.

If you don't want to wait 2 hours, you can invoke the function directly with:

$ serverless invoke -f retweet

If you want to remove the function from your AWS account you can simply use:

$ serverless remove

That's it. You now have your own Twitter bot able to retweeting and following users. You can learn how to add many more functionalities reading the Twython documentation.

Closing remarks

The full code used in this article is available on GitHub:

lorenzotenti / serverlessbot

serverlessbot

This is a Twitter bot retweeting at regular intervals the most relevant tweets about serverless computing.

Find it here.

A tutorial on how to build this bot can be found here.

It needs a config.py file with the Twitter API keys in order to work:

APP_KEY="YOUR_APP_KEY"
APP_SECRET="YOUR_APP_SECRET"
OAUTH_TOKEN="YOUR_OAUTH_TOKEN"
OAUTH_TOKEN_SECRET="YOUR_OAUTH_TOKEN_SECRET"

View on GitHub

The _serverlessbot_ built in this article can be found here.

I hope you liked this tutorial. If not, let me know why in the comments.