DEV Community: Deta

Building a Generative Image Bot with OpenAI, Telegram, FastAPI and Deta Space

Max — Fri, 31 Mar 2023 08:53:20 +0000

You may have seen some of the amazing images generated by new AI services, like Midjourney and Open AI's DALL·E 2.

Want to get in on the action? In this step-by-step guide, we'll build a personal Telegram bot that will let you use DALL·E 2 from your phone.

When we're done, you'll have a fully functional bot capable of creating unique images from your own input. Your server will run and your images will be stored in your own personal cloud on Deta Space, where you can easily access and use them in other projects.

Let's get going.

Prerequisites

Before starting, out you will need the following, at a minimum:

A Telegram Account
An OpenAI Account
A Deta Space Account (make sure to enable Developer Mode when signing up)

Optional:

Additionally, you’ll want Python on your local machine, if you want to develop locally

Architecture

Architecturally, the bot is pretty simple:

Interface: A personal Telegram bot will serve as your primary user interface, where you’ll send prompts and receive images.
Generative Image Service: DALL·E 2 (OpenAI’s service for turning language into images) will turn the prompts into images.
Server + State: In between, you’ll run a FastAPI server on Deta Space to bridge Telegram and DALL·E 2. This server will save every image in a personal file store on Deta Drive.

Step 1: Creating Your Bot and Getting the Keys

The first step is to create the bot in Telegram and get the required keys from both Telegram and Open AI.

Telegram Bot Token

Search for BotFather in Telegram and start a conversation with it, by clicking the START button.

Type /newbot, and follow the prompts to set up a new bot with a name and username for your bot. BotFather will give you the bot token that you will use to authenticate your bot. Save this token somewhere safe; you will need it later.

⚠️ Keep your token secure, it can be used anyone to control your bot.

Open AI API Key

Next, create an OpenAI account and get an OpenAI API key. This key will let your assistant generate an image with DALL·E 2.

Step 2: Writing Your Bot’s Code

The server we'll write is fairly simple and will do two core things:

Use incoming messages from Telegram to generate images with DALL·E 2.
Save the generated images to your personal cloud and send them back to Telegram.

The complete code is in this repository, and the core logic lives in the main.py file. The external Python libraries required are listed in the requirements.txt file.

Setup a Webhook with Telegram

Before Telegram can send our server a webhook on every message, it needs to know about our server. The /set_webhook route, when triggered, will configure the connection with Telegram. Once configured, Telegram will send events to our server's /open route.

The code assumes we’ve set the Bot Key as an environment variable in Deta Space (TELEGRAM). It also assumes it's running on Deta Space, as it grabs the server url from the Space environment. But we’ll deploy the code to Deta Space and set the environment variables later.

Let’s move onto the main logic of our bot.

Process Messages from Telegram

The first step for the core logic of the bot is to process an incoming message from Telegram as a prompt for the DALL·E 2 API. The /open route in our server will handle this.

One thing to be aware of is that anyone can message our bot. But we don’t want them to be able to generate and receive images. So we’ll create a simple system to control who has access by checking the chat_id Telegram sends. Later on, we'll create an admin page that will let us add chat_ids to an authorized list.

Here's the relevant code that parses the incoming request and checks the chat_id against a list of authorized chat_ids:



from deta import Base, Drive
# ... other code
CONFIG = Base("config")
# ... other code

@app.post("/open")
async def http_handler(request: Request):
    incoming_data = await request.json()

    if "message" not in incoming_data:
        print(incoming_data)
        return send_error(None, "Unknown error, lol, handling coming soon")

    prompt = incoming_data["message"]["text"]
    chat_id = incoming_data["message"]["chat"]["id"]

    # ... other functionality
    authorized_chat_ids = CONFIG.get("chat_ids")

    # ... other functionality
    if authorized_chat_ids is None or chat_id not in authorized_chat_ids.get("value"): 
        payload = {"text": "You're not authorized!", "chat_id": chat_id}
        message_url = f"{BOT_URL}/sendMessage"
        requests.post(message_url, json=payload).json()
        return

One note: we assume that we've stored a list of authorized chat_ids in a Deta Base, Deta's database service. After we validate the chat_id, we can grab the message and use it as a prompt for DALL·E 2 using a function get_image_from_prompt.

Generate Images with DALL-E 2

get_image_from_prompt is the function that sends the prompt to DALL·E 2, via the openai Python library. It'll get an image as a response.




OPEN_AI_KEY = os.getenv("OPEN_AI")

openai.api_key = OPEN_AI_KEY

def get_image_from_prompt(prompt):
    try:
        response = openai.Image.create(
            prompt=prompt,
            n=1,
            size="512x512",
            response_format="b64_json",
        )

        if "error" not in response:
            return {
                "b64img": response["data"][0]["b64_json"],  # type: ignore
                "created": response["created"],  # type: ignore
            }
        return {"error": response["error"]["message"]}  # type: ignore
    except Exception as e:
        return {"error": str(e)}  # type: ignore

Using the defaults we’ve set, we’ll get a 512x512 image response as base-64 encoded JSON. This code block assumes our server has the Open AI key from earlier, stored in its environment as OPEN_AI_KEY, but we’ll set that later.

Now that we've got an image, we can save it and send it back to Telegram!

Save the images and send them back to Telegram

To save our image, we'll use Deta Drive, which will let us create a file store in our personal cloud. We can then send it the image back to our Telegram chat.

Both are done in the rest of the /open handler function from earlier, using a helper function save_and_send_img:



from deta import Base, Drive
# ... other code

PHOTOS = Drive("generations") # instantiate the Deta Drive
# ... other code

def save_and_send_img(b64img, chat_id, prompt, timestamp):
    image_data = base64.b64decode(b64img)
    filename = f"{timestamp} - {prompt}.png"
    PHOTOS.put(filename, image_data)
    photo_payload = {"photo": image_data}
    message_url = f"{BOT_URL}/sendPhoto?chat_id={chat_id}&caption={prompt}"
    requests.post(message_url, files=photo_payload).json()
    return {"chat_id": chat_id, "caption": prompt}

# ... other code

@app.post("/open")
def http_handler(request: Request):
    # ... other code
    open_ai_resp = get_image_from_prompt(prompt)

    if "b64img" in open_ai_resp:
        return save_and_send_img(
            open_ai_resp["b64img"], chat_id, prompt, open_ai_resp["created"]
        )

    if "error" in open_ai_resp:
        return send_error(chat_id, open_ai_resp["error"])

    return send_error(chat_id, "Unknown error, lol, handling coming soon")

This function takes the base-64 encoded image and decodes it. Then it stores it to your Drive and sends it back to your Telegram bot.

Lastly, we'll talk about the management UI for your bot.

Creating a Management UI for your Bot

As the bot's owner, you'll want to know the bot's state, trigger the setup webhook with Telegram, and manage authorized chat_ids.

This is all done in a simple UI, declared in a single index.html file, with a bit of support from JS and CSS. This HTML file is served by the server in main.py via the /, /setup, and /authorize routes as a Jinja template.

Step 3: Running the Bot

Once you have written the code for your bot, you can deploy and run it out of your personal cloud in Deta Space.

Creating a Project in Deta Space

Before doing so, you’ll need to need to install and login to the Deta Space CLI. Then, ensure your local code has the server in your main.py file and dependencies in your requirements.txt file.

After you have the code, you can create your first Space Project. The CLI can bootstrap your project for Deta Space, if you haven't cloned the repository from GitHub.

In either case, you'll notice there is a Spacefile, which will tell Deta Space how to run your app. We'll want to ensure a few things about this Spacefile:

First, that the public_routes parameter is present, telling Deta Space to leave the /open route open to the public (so Telegram can send it events).
Second, to set two environment variables under the env parameter: TELEGRAM and OPEN_AI. We'll use these in our app to store the Telegram and Open AI keys, respectively.

The Spacefile also gives the app a name and sets an icon, among other things.

An example of a working Spacefile for our bot is here. But to make things easiest, it's recommended to clone the entire GitHub repository to bootstrap all the files.

Pushing Your Code and Setting Your Environment Variables

Once you’ve got your code ready and project created, you can push your local code to Deta Space with the Space CLI command space push. This will bundle your local code and deploy it as a usable app on Deta Space.

Once pushed, you’ll see a new icon in your Deta Space Canvas (the dashboard), which you'll see when you log in to Space. If you click the ..., then Settings, and then the Configuration tab, you’ll be able to set the environment variables.

Enter the Open AI API key and Telegram Bot Key from earlier and click Save.

Setting the Webhook & Chat ID

Now that you’ve added the environment variables, you can configure your assistant with the bot we’ve created in Telegram. Click your app’s icon from the Deta Space Canvas (dashboard) to visit the bot’s admin panel. (On Deta Space, you are authenticated, so this view is only accessible to you).

If you’ve saved the Configuration Variables from the last step, it should show you the button to setup your webhook. Click the Setup Webhook button to do so.

One final step we’ll have to do is add your own chat_id to the list of authorized users:

Add your bot in Telegram as a contact via its username.
Then talk to it with the command “/chatid” to get the ID of your chat from your bot.
Third, Visit the the admin UI again, click the authorize button and then the add new chat id button.
Finally, enter the ID into the input field and click submit.

You are now ready to generate images!

Running Your Assistant

Send your bot a message on Telegram, containing any prompt. Your bot should respond with the resulting image!

Accessing Your Photos

All the images you generate will be stored on your personal cloud in a Drive. You can view and download the photos, as well as program against them using the Deta Drive SDK. Click on the View Data option from the ... menu of your app, and then click the Drive tab to see everything you've got.

Other Options for Running Your Bot

You can install the app built in this tutorial in your own personal cloud in one click here. No coding is necessary, you’ll just need to:

create a Telegram bot.
set the Open AI and Telegram bot keys.
visit the “/” route to configure everything before using the bot.

You can also run your assistant anywhere a uvicorn server will run, including locally for development. In the local case, you’ll need to use something like ngrok to tunnel your local host to an https url (and modify the DETA_SPACE_HOSTNAME variable in the code to grab the nrgok hostname).

Final Words

Congratulations — you’ve got an AI assistant running in that can be accessed from your phone. It'll operate and store everything in your own cloud.

One idea for you to play with is to expose the image-generating function as an API, which you can use from a number of other apps (check out the API Keys on Deta Space).

But have fun generating all sorts of images. In the next part of our series, we’ll be adding the ability to chat with our assistant, similar to ChatGPT — stay tuned!

Special thanks to Bighneswar Parida for assistance with code + words.

Get started with FastAPI JWT authentication – Part 2

Rohan — Thu, 22 Apr 2021 14:52:59 +0000

This is the second of a two part series on implementing authorization in a FastAPI application using Deta. In the previous article, we learned a bit about JWT, set up the project, and finished the building blocks of authorization logic. In this article, let's implement the logic, and deploy our app on Deta micros! The full code is available here.

Implementing the auth logic

Before we implement the auth logic, let's create a data model for login and signup details.

In user_model.py :

from pydantic import BaseModel

class AuthModel(BaseModel):
    username: str
    password: str

This model represents the data that we can expect from the client when they hit /login or /signup endpoints.

Update the main.py , with the following import statements

from auth import Auth
from user_model import AuthModel
from fastapi import FastAPI, HTTPException, Security
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer

Also, we will create an auth_handler to access the logic from the Auth class. We will use security in our protected endpoints to access the token from the request header.

security = HTTPBearer()
auth_handler = Auth()

Here is the code for /signup endpoint.

@app.post('/signup')
def signup(user_details: AuthModel):
    if users_db.get(user_details.username) != None:
        return 'Account already exists'
    try:
        hashed_password = auth_handler.encode_password(user_details.password)
        user = {'key': user_details.username, 'password': hashed_password}
        return users_db.put(user)
    except:
        error_msg = 'Failed to signup user'
        return error_msg

In this function we are checking if a user with the username already exists in our users_db. If so, we can simply return a message indicating that the account already exists. If the user doesn't already exists we can hash the password using the encode_password function from auth.py and store the user in users_db. In case of any errors while adding the user to the base, we can return a failure message.

/login endpoint is pretty simple. This also takes in the argument user_details , which has the username and password.

@app.post('/login')
def login(user_details: AuthModel):
    user = users_db.get(user_details.username)
    if (user is None):
        return HTTPException(status_code=401, detail='Invalid username')
    if (not auth_handler.verify_password(user_details.password, user['password'])):
        return HTTPException(status_code=401, detail='Invalid password')

    access_token = auth_handler.encode_token(user['key'])
    refresh_token = auth_handler.encode_refresh_token(user['key'])
    return {'access_token': access_token, 'refresh_token': refresh_token}

If the account with the username doesn't exist, or if the hashed password in the users_db doesn't match the input password we can simply raise an HTTPException. Otherwise, we can return access_token and refresh_token.

@app.post('/secret')
def secret_data(credentials: HTTPAuthorizationCredentials = Security(security)):
    token = credentials.credentials
    if(auth_handler.decode_token(token)):
        return 'Top Secret data only authorized users can access this info'

@app.get('/notsecret')
def not_secret_data():
    return 'Not secret data'

The /secret endpoint only returns the "Secret Data" if the token argument is valid. However, if the token is invalid or an expired token, then decode_token raises a HTTPException. The token is usually passed in the request header as Authorization: Bearer <token>. Therefore, to get the token we can wrap the input credentials around HTTPAuthorizationCredentials tag. Now we can access the token from the request header in credentials.credentials.

The /not_secret endpoint is an example of an unprotected endpoint, which doesn't require any authentication.

@app.get('/refresh_token')
def refresh_token(credentials: HTTPAuthorizationCredentials = Security(security)):
    refresh_token = credentials.credentials
    new_token = auth_handler.refresh_token(refresh_token)
    return {'access_token': new_token}

/refresh_token endpoint is also pretty simple, it receives a refresh token which is then passed onto the the function from auth logic to get a new token.

Here is a look at main.py at the end:

from fastapi import FastAPI, HTTPException, Security
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from auth import Auth
from deta import Deta
from user_model import AuthModel

deta = Deta()
users_db = deta.Base('users')

app = FastAPI()

security = HTTPBearer()
auth_handler = Auth()

@app.post('/signup')
def signup(user_details: AuthModel):
    if users_db.get(user_details.username) != None:
        return 'Account already exists'
    try:
        hashed_password = auth_handler.encode_password(user_details.password)
        user = {'key': user_details.username, 'password': hashed_password}
        return users_db.put(user)
    except:
        error_msg = 'Failed to signup user'
        return error_msg

@app.post('/login')
def login(user_details: AuthModel):
    user = users_db.get(user_details.username)
    if (user is None):
        return HTTPException(status_code=401, detail='Invalid username')
    if (not auth_handler.verify_password(user_details.password, user['password'])):
        return HTTPException(status_code=401, detail='Invalid password')

    access_token = auth_handler.encode_token(user['key'])
    refresh_token = auth_handler.encode_refresh_token(user['key'])
    return {'access_token': access_token, 'refresh_token': refresh_token}

@app.get('/refresh_token')
def refresh_token(credentials: HTTPAuthorizationCredentials = Security(security)):
    refresh_token = credentials.credentials
    new_token = auth_handler.refresh_token(refresh_token)
    return {'access_token': new_token}

@app.post('/secret')
def secret_data(credentials: HTTPAuthorizationCredentials = Security(security)):
    token = credentials.credentials
    if(auth_handler.decode_token(token)):
        return 'Top Secret data only authorized users can access this info'

@app.get('/notsecret')
def not_secret_data():
    return 'Not secret data'

To test the app, go to the terminal in the same directory and run uvicorn main:app, you can then go /docs on the local endpoint (for me it was http://127.0.0.1:8000/docs) to test the application.

/signup

curl -X 'POST' \
  'http://127.0.0.1:8000/signup' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "username": "flyingsponge",
  "password": "SpongePassword"
}'

Response Body
{
  "key": "flyingsponge",
  "password": "$2b$12$Ml66gTN0j4sAOkoDJ1aKnOmP0ye1FBNNk1QzEt0/6LgemgOUj469e"
}

/login

curl -X 'POST' \
  'http://127.0.0.1:8000/login' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "username": "flyingsponge",
  "password": "SpongePassword"
}'

Response Body
{
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTkwMTY4MzgsImlhdCI6MTYxOTAxNTAzOCwic3ViIjoiZmx5aW5nc3BvbmdlIn0.XnPaDwmj30M3vMOaPUOrqESIBNx0mctbjNW5jY8hIjQ",
  "refresh_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTkwNTEwMzgsImlhdCI6MTYxOTAxNTAzOCwic3ViIjoiZmx5aW5nc3BvbmdlIn0.Pm68HQFM6NwUxnEjFUxxTiOYT3KBqchD_e2g0LfP5Co"
}

/secret

curl -X 'POST' \
  'http://127.0.0.1:8000/secret' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTkwMTY2MTIsImlhdCI6MTYxOTAxNDgxMiwic3ViIjoicmFta2kifQ.U8fUN1x1Gz18f7oA_y7Z9DE-1FIH9Ps0sDilwORvmI8' \
  -d ''

Response body
"Top Secret data only authorized users can access this info"

/notsecret

curl -X 'GET' \
  'http://127.0.0.1:8000/notsecret' \
  -H 'accept: application/json'

Response Body
"Not secret data"

/secret

curl -X 'POST' \
  'http://127.0.0.1:8000/secret' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTg5MzU3MzcsImlhdCI6MTYxODkzNTY3Nywic3ViIjoicm9oYW4ifQ.dja0E6SUaZfEvYVKySjLE9OLXOtob5pjpy3R_rlCD7c' \
  -d ''
Response Body
{
  "detail": "Token expired"
}

Now that the token is expired, let's get a new one

/refresh_token

curl -X 'GET' \
  'http://127.0.0.1:8000/refresh_token' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTkwNTA3NzYsImlhdCI6MTYxOTAxNDc3Niwic3ViIjoicmFta2kifQ.2J0O4RKwEcABxe6hEX7ZshMo66J6D0dD6-hYeFLBFGg'
Response Body
{
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTkwMTY2MTIsImlhdCI6MTYxOTAxNDgxMiwic3ViIjoicmFta2kifQ.U8fUN1x1Gz18f7oA_y7Z9DE-1FIH9Ps0sDilwORvmI8"
}

Deploy on Deta micros

Before we begin, make sure to install the Deta CLI. After installing, run the following commands in the same directory to deploy our app on Deta micros.

deta login

We also need to add a .env file with the secret.

APP_SECRET_STRING=SECRET_STRING

Now we need to update our micro by doing:

deta new 
deta update -e .env
deta deploy
deta visor disable

Summary

Our simple FastAPI application with JWT auth is now ready! As you can probably tell, we are not doing anything "secret" with our authorization. This article is just a template for implementing authorization. You can build on this template to build a fullstack application that relies on authorization. The full code is available here.

Link to the first article!

Get started with FastAPI JWT authentication – Part 1

Rohan — Mon, 12 Apr 2021 23:26:56 +0000

This is the first of a two part series on implementing authorization in a FastAPI application using Deta. In this article, we will learn about JWT tokens, set up the project, and build the auth logic. In the next article, we will implement the auth logic in a FastAPI application. The full code is available here.

Introduction

Implementing authorization can be useful, as it provides the client access to a specific set of functions, actions, data, etc. Consider an e-commerce website, you would want to make sure users are authorized before they can look at items in the cart. Another example is a chat application where only the owner has the right to add/remove people.

JWT (or JSON web tokens) are simply base64 strings that encode some information about the client. These tokens are signed using a secret key or a public/private key. We will implement the former method. Essentially, when the client is logged in, the server sends back a response with a signed token. Subsequently, the client can send requests to the server with the token as a header to access authorized routes, data, functions etc.

Let's get started

Agenda

Setup
FastAPI app skeleton + Auth logic

Setup

Tools

FastAPI: we are using FastAPI to build the application
Deta Base (Base) : database for our application
Deta Micro: host our application
pyjwt: library for encoding and decoding JWT tokens
passlib[bcrypt]: password hashing library

Install

To get started, create a folder for this project fastapi-jwt , and create a requirements.txt file with the following lines:

deta
fastapi
uvicorn
pyjwt
passlib[bcrypt]

Run the following command to install the libraries

pip install -r requirements.txt

Before we begin with the project we also need to get a Deta project key to use with Deta Base. We are using Base to store user account information such as username and hashed password.

To do that, navigate to the Deta Console then click on the arrow on the top left.
If you don't already have a Deta account, create one for free. Once you confirm your email, Deta will automatically generate a Project Key, this is the one we need, copy it and store it securely.

Create a new project and make sure to save the key in a secure place!

Add the key to your environment variables like this DETA_PROJECT_KEY=YOUR_COPIED_PROJECT_KEY

That's it for the setup, we have everything we need to get rolling. Let's go!

FastAPI app skeleton and Auth logic

Here is how our folder structure will look like at the end:


fastapi-jwt/
    ├── main.py
    ├── auth.py
    ├── user_model.py
    └── requirements.txt

In main.py , let's set up our FastAPI application, Deta Base, and skeletons for all the endpoints.

from fastapi import FastAPI, HTTPException
from deta import Deta

deta = Deta()
users_db = deta.Base('users')

app = FastAPI()

@app.post('/signup')
def signup():
    return 'Sign up endpoint'

@app.post('/login')
def login():
    return 'Login user endpoint'

@app.get('/refresh_token')
def refresh_token():
    return 'New token'

@app.post('/secret')
def secret_data():
    return 'Secret data'

@app.get('/notsecret')
def not_secret_data():
    return 'Not secret data'

users_db is our base where we store the account's hashed password. The schema for users will look like the following:

{
    key: str, # username
    encoded_password: str
}

Now let's head over to auth.py, to handle the authentication logic:

import os
import jwt # used for encoding and decoding jwt tokens
from fastapi import HTTPException # used to handle error handling
from passlib.context import CryptContext # used for hashing the password 
from datetime import datetime, timedelta # used to handle expiry time for tokens

class Auth():
    hasher= CryptContext(schemes=['bcrypt'])
    secret = os.getenv("APP_SECRET_STRING")

    def encode_password(self, password):
        return self.hasher.hash(password)

    def verify_password(self, password, encoded_password):
        return self.hasher.verify(password, encoded_password)

So far, we just imported all the tools from the libraries, and we created the Auth class with two functions. We don't want to store the plain text password in our users Base. Therefore, we can use the encode_password function to encode the password using the passlib['bcrypt'] library. We can store this encoded password in our users_db base when the user makes an account.

We also have another function verify_password which checks if the plain password and the encoded password from users_db match. This can be useful to verify user in the /login endpoint.

Notice that we get the variable secret from our environment, make sure to generate a long secure string and store it in your environment variables under the name APP_SECRET_STRING.

Now that we have a way to verify passwords, and hash passwords, it is time to handle the logic for encoding and decoding JSON web tokens. The tokens are the essence of auth logic.

Inside the Auth class, add the following functions.

    def encode_token(self, username):
        payload = {
            'exp' : datetime.utcnow() + timedelta(days=0, minutes=30),
            'iat' : datetime.utcnow(),
        'scope': 'access_token',
            'sub' : username
        }
        return jwt.encode(
            payload, 
            self.secret,
            algorithm='HS256'
        )

    def decode_token(self, token):
        try:
            payload = jwt.decode(token, self.secret, algorithms=['HS256'])
            if (payload['scope'] == 'access_token'):
                return payload['sub']   
            raise HTTPException(status_code=401, detail='Scope for the token is invalid')
        except jwt.ExpiredSignatureError:
            raise HTTPException(status_code=401, detail='Token expired')
        except jwt.InvalidTokenError:
            raise HTTPException(status_code=401, detail='Invalid token')

The encode_token function takes a username as a parameter and uses pyjwt to create an access_token. We are using timedelta to set the expiry of the token for 30 mins. The scope parameter can be used to verify that the client uses an access_token. We can use this function inside the /login endpoint, and return a token to the client.

decode_token takes a token as a parameter, and attempts to decode it using the secret. If there are any errors like expired token or an invalid token, we can simply raise an HTTPException. Otherwise, we can return the username. This will be helpful to us when the client interacts with protected data, functions, etc. We can use this function to simply verify if they have access to the response.

When the token expires, the application forces the user to login. To avoid this, we can create two tokens access_token and refresh_token when the user logins. The refresh token usually has a longer expiry time than the access_token, and will only be used to create a new token. Everytime the access_token expires, the client sends a request to the server to create a new access_token using the refresh_token. If the refresh_token expires then the client will be forced to login.

We need two more function to handle the refresh_token logic.

    def encode_refresh_token(self, username):
        payload = {
            'exp' : datetime.utcnow() + timedelta(days=0, hours=10),
            'iat' : datetime.utcnow(),
        'scope': 'refresh_token',
            'sub' : username
        }
        return jwt.encode(
            payload, 
            self.secret,
            algorithm='HS256'
        )
    def refresh_token(self, refresh_token):
        try:
            payload = jwt.decode(refresh_token, self.secret, algorithms=['HS256'])
            if (payload['scope'] == 'refresh_token'):
                username = payload['sub']
                new_token = self.encode_token(username)
                return new_token
            raise HTTPException(status_code=401, detail='Invalid scope for token')
        except jwt.ExpiredSignatureError:
            raise HTTPException(status_code=401, detail='Refresh token expired')
        except jwt.InvalidTokenError:
            raise HTTPException(status_code=401, detail='Invalid refresh token')

As you can tell, we are setting the expiry time of the refresh_token to be 10 hours which is more than the access_token. Plus, we are simply using refresh_token to create a new access_token. The scope parameter from JWT token ensures that the refresh_token is used only for creating new tokens, and access_token is used only for interacting with protected endpoints.

That is all we need for the auth logic! Here is what the file looks like at the end:

auth.py

import os
import jwt # used for encoding and decoding jwt tokens
from fastapi import HTTPException # used to handle error handling
from passlib.context import CryptContext # used for hashing the password 
from datetime import datetime, timedelta # used to handle expiry time for tokens

class Auth():
    hasher= CryptContext(schemes=['bcrypt'])
    secret = os.getenv("APP_SECRET_STRING")

    def encode_password(self, password):
        return self.hasher.hash(password)

    def verify_password(self, password, encoded_password):
        return self.hasher.verify(password, encoded_password)

    def encode_token(self, username):
        payload = {
            'exp' : datetime.utcnow() + timedelta(days=0, minutes=30),
            'iat' : datetime.utcnow(),
        'scope': 'access_token',
            'sub' : username
        }
        return jwt.encode(
            payload, 
            self.secret,
            algorithm='HS256'
        )

    def decode_token(self, token):
        try:
            payload = jwt.decode(token, self.secret, algorithms=['HS256'])
            if (payload['scope'] == 'access_token'):
                return payload['sub']   
            raise HTTPException(status_code=401, detail='Scope for the token is invalid')
        except jwt.ExpiredSignatureError:
            raise HTTPException(status_code=401, detail='Token expired')
        except jwt.InvalidTokenError:
            raise HTTPException(status_code=401, detail='Invalid token')

    def encode_refresh_token(self, username):
        payload = {
            'exp' : datetime.utcnow() + timedelta(days=0, hours=10),
            'iat' : datetime.utcnow(),
        'scope': 'refresh_token',
            'sub' : username
        }
        return jwt.encode(
            payload, 
            self.secret,
            algorithm='HS256'
        )
    def refresh_token(self, refresh_token):
        try:
            payload = jwt.decode(refresh_token, self.secret, algorithms=['HS256'])
            if (payload['scope'] == 'refresh_token'):
                username = payload['sub']
                new_token = self.encode_token(username)
                return new_token
            raise HTTPException(status_code=401, detail='Invalid scope for token')
        except jwt.ExpiredSignatureError:
            raise HTTPException(status_code=401, detail='Refresh token expired')
        except jwt.InvalidTokenError:
            raise HTTPException(status_code=401, detail='Invalid refresh token')

In the next article, we will implement the logic in a FastAPI application and deploy our app on Deta micros! The full code is available here.

Persistent To Dos with Next.js + Deta Base in 7 minutes

Max — Wed, 11 Nov 2020 08:29:03 +0000

Next.js adds a lot on top of React; with support for api routes with serverless functions out of the box, Next let's you do traditional 'server side' tasks, like making authenticated requests to a database. If you deploy on Vercel, the pages/api directory will auto deploy as functions.

As we spoke about previously, traditional databases are not a great fit in the serverless model, where persistent connections don't mesh well with asynchronous, ephemeral, functions; Vercel suggests connection pooling as one way to mitigate these issues. Using a pure serverless database—where database requests do not rely on a persistent database connection—is another way around this issue.

This tutorial will guide you through creating a To Do app using Next.js and Deta Base, with deployments on Vercel. This app will be fundamentally different from a client side state model where To Do state is only stored in a React Component. In this app, the serverless functions will talk to Deta Base which will store the To Do state. This will provide To Dos a persistence that extends beyond component unmount and, as will be seen, Deta Base's GUI can be used to update To Do state, feeding back into our Next.js app.

This app uses the Create Next App starter, and the full source code is here.

Deploy instructions are here.

Design

The fundamental unit of our application will be a To Do, which will exist as a JSON object:

{
    "content": "Wake Up On Time", // string
    "isCompleted": false // boolean
}

These to dos will be stored in Deta Base and ultimately rendered by our Next.js app. To do so requires adding the deta dependency to your project using npm install deta or yarn add deta.

Additionally, our Next.js app needs to be able to generate and interact with this data. We can tie the four basic CRUD functions to two endpoints / serverless functions in Next.js

Create a new To Do: POST api/todos
Read all the To Dos: GET api/todos
Update a To Do (of id tid ): PUT api/todos/{tid}
Delete a To Do (of id tid ): DELETE api/todos/{tid}

The basic Next.js file structure for our application is as follows (modified from the Create Next App starter).

/pages
    index.js (our frontend logic)
    /api
            /todos
                index.js (function, will handle the GET & POST)
                [tid].js (function, will handle the PUT & DELETE)

Creating a To Do

To create a To Do, let's create an api call that will call POST api/todos based on some newContent stored in a React State Hook (this is tied to an input element in line 84):

export default function Home() {

  const [newContent, setNewContent] = useState('');

  ...


  const createToDo = async () => {
    const resp = await fetch('api/todos', 
      {
        method: 'post', 
        body: JSON.stringify({content: newText})
      }
    );
    // await getToDos(); To Be Implemented
  }

    ...
    return (
    ...
            <input className={styles.inpt} onChange={e => setNewContent(e.target.value)}></input>
    ...
    )

}

The function createToDo, when called, will pull the value of newContent from state in React and POST it to our endpoint, which we handle at pages/api/todos/index.js (link here):

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import { Deta } from 'deta';

const deta = Deta(process.env.DETA_PROJECT_KEY);

const base = deta.Base('todos');

export default async (req, res) => {
  let { body, method } = req;
  let respBody = {};

  if (method === 'GET') {

    // To Be Implemented

  } else if (method === 'POST') {

    body = JSON.parse(body);
    body.isCompleted = false;
    respBody = await base.put(body);
    res.statusCode = 201;

  }

  res.json(respBody);
}

In this handler, we access a project key that we get from Deta and store in a Vercel Environment Variable. This key allows us to talk to any Base in that Deta project, in this case a database we have called todos. Using the Deta SDK, we can take the content from the api call, add an isCompleted field, and use the put method to store our new to do in our database. A key will be automatically generated under which this item will be stored.

Reading To Dos

To read all our To Dos, let's create an api call that will call GET api/todos and store it in a React hook in the home component of pages/index.js .

Secondly, let's also use a React useEffect hook to call this function when our component mounts.

Third, let's create two lists from our to dos, that will give us the list of to dos by completion status, which we will display in different parts of our app (lines 89 and 106 of index.js).

This relies on us having a working ToDo component, which we will assume correctly displays content and completion status for now.

export default function Home() {

  const [newContent, setNewContent] = useState('');

  const [toDos, setToDos] = useState([]);

  const getToDos = async () => {
    const resp = await fetch('api/todos');
    const toDos = await resp.json();
    setToDos(toDos);
  }

    ...

  useEffect(() => {
    getToDos();
  }, [])

  const completed = toDos.filter(todo => todo.isCompleted);
  const notCompleted = toDos.filter(todo => !todo.isCompleted);

    ...

  return (

    ...

     <div className={styles.scrolly}>
        {notCompleted.map((todo, index) => 
          <ToDo 
            key={todo.key} 
            content={`${index + 1}. ${todo.content}`} 
            isCompleted={todo.isCompleted} 
            // onChange={() => updateToDo(todo)} To Be Implemented
            // onDelete={() => deleteToDo(todo.key)} To Be Implemented
          />
        )}
     </div>

    ...

     <div className={styles.scrolly}>
       {completed.map((todo, index) => 
         <ToDo 
           key={todo.key} 
           content={`${index + 1}. ${todo.content}`} 
           isCompleted={todo.isCompleted}
           // onChange={() => updateToDo(todo)} To Be Implemented
           // onDelete={() => deleteToDo(todo.key)} To Be Implemented
         />
       )}
    </div>

    ...

    )

}

The serverless function handler in pages/api/todos/index.js looks as follows:

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import { Deta } from 'deta';

const deta = Deta(process.env.DETA_PROJECT_KEY);

const base = deta.Base('todos');

export default async (req, res) => {
  let { body, method } = req;
  let respBody = {};

  if (method === 'GET') {

    const {value: items} = await base.fetch([]).next();
    respBody = items;
    res.statusCode = 200;

  }

...

  res.json(respBody);
}

Here the GET request is handled in the function, using a Deta Base's fetch to return all items in a database called todos .

Updating a To Do

To update a To Do's completion status, we create a function updateToDo that will call PUT api/todos/{tid} based on our ToDo component triggering an onChange function (which is implemented by a checkbox being checked / unchecked):

export default function Home() {

    ...
    const updateToDo = async (todo) => {
            let newBody = { 
               ...todo,
               isCompleted: !todo.isCompleted
            };
            const resp = await fetch(`api/todos/${todo.key}`, 
               {
                   method: 'put', 
                   body: JSON.stringify(newBody)
               }
            );

            await getToDos();
        }
    ...
    return (
    ...

            <ToDo 
                key={todo.key} 
                content={`${index + 1}. ${todo.content}`} 
                isCompleted={todo.isCompleted} 
                onChange={() => updateToDo(todo)}
        />
    ...
    )
}

The function will send a PUT to with the opposite pages/api/todos/[tid].js :

import { Deta } from 'deta';

const deta = Deta(process.env.DETA_PROJECT_KEY);

const base = deta.Base('todos');

export default async (req, res) => {

  let { body, method, query: { tid } } = req;
  let respBody = {};

  if (method === 'PUT') {

    body = JSON.parse(body);
    respBody = await base.put(body);
    res.statusCode = 200;

  } else if (method === 'DELETE') {

    // To Be Implemented

  }

  res.json(respBody);
}

In this handler, we pass the unchanged body through our put method to store our updated to do in our database. Because the body contains the key this will correctly overwrite the old record.

Deleting a To Do

Finally, to delete a To Do, let's add the api call that will call DELETE api/todos/{tid} based on a button click:

export default function Home() {

  ...


  const deleteToDo = async (tid) => {
    const resp = fetch(`api/todos/${tid}`, {method: 'delete'});
    setTimeout(getToDos, 200);
  }

    ...
    return (
    ...
                <ToDo 
                  key={todo.key} 
                  content={`${index + 1}. ${todo.content}`} 
                  isCompleted={todo.isCompleted} 
                  onChange={() => updateToDo(todo)}
                  onDelete={() => deleteToDo(todo.key)}
                />  
    ...
    )

}

The function deleteToDo, when called, will make a DELETE request to pages/api/todos/{tid} , whose handler looks as follows:

import { Deta } from 'deta';

const deta = Deta(process.env.DETA_PROJECT_KEY);

const base = deta.Base('todos');

export default async (req, res) => {

  let { body, method, query: { tid } } = req;
  let respBody = {};

  if (method === 'PUT') {

  ...

  } else if (method === 'DELETE') {

    respBody = await base.delete(tid);
    res.statusCode = 200;

  }

  res.json(respBody);
}

In this handler, we pass use the delete method from the Deta SDK.

Final Things

All the logic is implemented at this point and you can deploy the resulting application yourself to Vercel.

You can also do it in a few clicks: just grab a Deta project key, click the button below, and set the project key as an environment variable-- DETA_PROJECT_KEY--during Vercel's flow.

We can't forget to mention that you can now view and manage your to dos from Deta Base's GUI, Guide. If you add or modify one of your To Dos from here, the changes will load in the Vercel app on page refresh.

The last thing worth mentioning is that this app uses a standard vanilla React pattern for managing the application state to keep things simple. However, we can take advantage of some smart things Next enables (in tandem with libraries like useSWR) to improve performance. If you've deployed this app, you'll notice the delays on create, modification and deletion, as the serverless functions take around 300ms to respond. With some improvements, we can boost performance and create a feeling of an instant response on the client side. Stay tuned for round 2.

Choosing the Right Database for Serverless Applications

Aavash Shrestha — Wed, 28 Oct 2020 13:38:22 +0000

Serverlesss computing, where compute instances are provisioned and destroyed on the fly, ditches the idea of long running servers. Traditional databases were designed primarily for self-managed long running servers. This has lead to various problems in using traditional databases with serverless applications:

Provisioning, Scaling and Maintenance

The main idea of serverless is to offload provisioning, scaling, capacity planning and maintenance of servers, and focus on application development.

This conflicts directly with how traditional databases are set up. You have to provision a fixed amount of servers (which do not scale down, you might under-use them), configure scaling policies based on predictions, maintain backups, install updates, and ensure robustness and security.

Persistent Connections

Traditional database clients maintain a persistent connection to the database server and all requests are made on this connection. The set up and tear down of this connection are costly processes. As a result, setting up and maintaining persistent connections every time a new compute instance spins up becomes very inefficient for serverless applications.

Number of Connections

Serverless applications may scale up to hundreds or thousands of concurrent compute instances, which means database servers have to maintain as many connections as number of instances.

This is hard on performance, as a traditional database becomes more inefficient as the number of connections increases, while there is also a limit in the number of connections a single database server can keep open.

Global Access

Serverless compute has made it easier to provide low-latency access to your applications all over the world. However, replicating data and maintaining data consistency across multiple regions is a difficult challenge for traditional databases.

The Solution

Serverless applications require a suitable database designed to meet the needs of the serverless model. The right database should:

offer a fully-managed solution; no servers to provision and maintain, on demand scaling (up and down), automated backups, easy recovery, and high security
offer an API that does not depend on costly persistent connections, while still remaining fast
scale efficiently with the number of serverless compute instances
offer global data replication and consistency with low-latency

As serverless computing sees more adoption, more developers will need a suitable database for their serverless needs and move on from implementing workarounds with traditional databases.

And Lastly

Our team at Deta is working hard in building a database that overcomes the inadequacies of traditional databases in serverless applications with focus on end-user simplicity with Deta Base. You can get started here. We highly appreciate any sort of feedback.

Introducing Base UI: a GUI for your Deta Base

Max — Wed, 30 Sep 2020 15:44:58 +0000

What's Base UI?

Base UI (beta) is a graphical user interface for Deta Base. Base UI was inspired by the struggles of many developers in interacting with the data powering their apps, outside the confines of the app itself.

A video tour is available at the end of the post.

Motivation

Since shipping the earliest alpha of Deta & Deta Base, devs have resonated with the simplicity of spinning up a database using Deta. In it's purest form, there is no marginal config step required, it's just code.

This approach is great when you want to design state from the comforts of your text editor. While Base has provided a persistent state solution for some inspirational apps so far, we realized it was falling short on another dimension we didn't initially expect.

The biggest shortcoming we've heard from users about Base is the lack of a UI. We often heard 'I want to be able to easily see and interact with the data my app generates'. We noticed some users would hack around this shortcoming, adding auxiliary logic to their application or a side script to quickly inspect or modify the records their main app generates.

But going through both a text editor and running a script to inspect and modify records is less than ideal. Two members of our community— turned summer fellows—both took it a step further, each hacking their own Base records viewers together (first w/ source; second w/ source).

Suffice to say, we are extremely excited to address this need and launch Deta's official Base GUI solution into beta, building off the work of the second solution. With Base Guide, you can rapidly inspect, add, delete, and modify the records contained in your Base.

Inspecting Records

To view a Base's records, all you need to do is open an individual Base from the Deta Project Sidebar. By default the Base's records will load into Base UI.

If you want to inspect a slice of the data within a Base, you can use Base's queries to do so, i.e.:

[{"cuisine": "Italian", "have_visited": false}]

Adding Records

To add records to a base, click the '+ Add' button and a new row will appear at the top of your Base's UI.

The new row will be treated as a candidate edit, with a yellow background, which can be permanently saved by clicking the 'Save' button.

Modifying Records

For primitive types (strings, booleans, and numbers), you can modify any record directly from within a cell.

For advanced editing, i.e. to change the type of a given value, or edit arrays or objects, you can expand the cell and do so.

Once you are satisfied with your edits, simply click the 'Save Edits' button, as before, to modify these records in your Base. If you want to discard all your edits locally, click 'Undo Changes'.

Deleting Records

For every record displayed in Base UI, there is a checkbox on the left side. By clicking the checkbox, the record will be highlighted in red as a delete candidate. To go forward with the deletion, simply click the 'Delete' item.

For candidate adds, edits, and deletes, the 'Undo' button will revert the Base Guide to the state that was last pulled.

Start with Base UI Today

To get started with Base UI, simply log-in / sign-up with Deta, grab a project key, and create a Deta Base in your Python or Node.js app. Deta Guide will be accessible for any Base you create.

Tour

Thanks

We'd love to thank the Deta community for the incredible feedback that led to this initiative as well as Samarth Jajoo for the initial implementation which we built on top of!

Track custom github repository metrics with Github Webhooks, FastAPI and DETA

Aavash Shrestha — Mon, 28 Sep 2020 11:43:04 +0000

Tracking important metrics for a Github repository facilitates analysis and optimization of a team's development and delivery process. The metrics help in identifying bottlenecks, prioritizing resources and setting goals.

In this tutorial, we are going to implement and deploy a simple self-hosted application that keeps track of the Code review turnaround, the time take between a code review assignment and completion, for a repository using Github Webhooks, FastAPI and Deta.

This tutorial showcases how, with Deta, you can direct most of your focus and effort on developing the tool and get it seamlessly deployed to production.

The application's users can either generate a graph or get a json response of average code review turnarounds for a specified duration averaged over a specified period. To generate the graph, we will use Bokeh, an interactive visualization library.

{
  "2020-09-18T09:00:22": 0, 
  "2020-09-19T09:00:22": 454.88,
  "2020-09-20T09:00:22": 1315.75,
  "2020-09-21T09:00:22": 87.13,
  "2020-09-22T09:00:22": 178.05,
  "2020-09-23T09:00:22": 95.83,
  "2020-09-24T09:00:22": 40.7,
  "2020-09-25T09:00:22": 0
}

I have named the application GRT (Github Code Review Turnaround). A complete source code of the application is available on github.

Application Design

GRT needs to achieve three main things in order to keep track of the code review turnarounds and allow users to easily retrieve them:

Know when a code review request has been requested or deleted, and when a code review has been submitted. We use Github Webhooks for this.
Store and update information about code reviews in a persistent storage. For this we use Deta Base.
Offer an api for users to see the average code review turnaround. The user should be able to specify the type of response, duration and the period to average the metrics over.

Implementation and Deployment

The following guide assumes that you have signed up for Deta and have the Deta CLI installed.

This guide is also only for unix environments. Some shell commands might be different for windows. Please, use respective commands for windows.

Create a deta micro

Firstly, we create a new FastAPI application on Deta.

Create a directory called grt and cd into it.

$ mkdir grt && cd grt

Create two files main.py and requirements.txt in the root of the directory.

$ touch main.py requirements.txt

Specify fastapi and bokeh as dependencies in the requirements.txt file. It should look like this:

fastapi
bokeh

Create the Deta Micro. With a main.py file and fastapi specified as a dependency in the requirements.txt file, all you need to do is type deta new in order to create a new fastAPI app. From the root of the directory, enter

$ deta new
Successfully created a new micro
{
    "name": "grt",
    "runtime": "python3.7",
    "endpoint": "https://{your_subdomain}.deta.dev",
    "visor": "enabled",
    "http_auth": "enabled"
}
Adding dependencies...
Collecting fastapi
...
Collecting bokeh
...
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 bokeh-2.2.1 fastapi-0.61.1 numpy-1.19.2 packaging-20.4 pillow-7.2.0 pydantic-1.6.1 pyparsing-2.4.7 python-dateutil-2.8.1 six-1.15.0 starlette-0.13.6 tornado-6.0.4 typing-extensions-3.7.4.3

The endpoint will be different for your micro.

You should see that http_auth is enabled by default. We will disable the auth for the github webhook and use a webhook secret.

Set up the webhook

The command deta details shows details about your deployed micro including your micro's http endpoint. Copy your micro's endpoint from the output of deta details.

$ deta details
{
    "name": "grt",
    "runtime": "python3.7",
    "endpoint": "https://{your_subdomain}.deta.dev",
    "visor": "enabled",
    "http_auth": "enabled"
}

Go to Webhooks under Settings for the repository you want to track your metrics on and click on Add Webhook.
In the Payload URL, use your micro's endpoint with the route /webhook_events as the webhook endpoint.

https://{your_subdomain}.deta.dev/webhook_events

Change the Content type to application/json.
Generate a long secure random string (there are services online that do this) and use that as the Webhook Secret. Keep hold of this secret as you will need it to set up the app's environment later.
Select Let me select individual events when selecting the events to trigger the webhook. Select the following events:
- Pull requests : To know when a code review is requested
- Pull requests reviews : To know when a code review has been submitted or deleted
Click on Add Webhook to add the webhook.

Set up the environment

The webhook secret used in setting up the webhook is provided to the micro through an environment variable WEBHOOK_SECRET.

Create a .env file in the app's root directory and add your secret in the file. Make sure not to expose this file publicly.

$ echo WEBHOOK_SECRET=your_webhook_secret > .env
$ cat .env
WEBHOOK_SECRET=your_webhook_secret

Update the environment variables of your app.

$ deta update -e .env

You should see that the environment variables have been successfully updated.

Implement the webhook endpoint

Time to code, let's add a POST endpoint that receives webhook events from github.

Open main.py in your editor and add the following code.

from fastapi import FastAPI, Request

# FastAPI app
app = FastAPI()

@app.post("/webhook_events")
async def webhook_handler(request: Request):
    # handle events
    payload = await request.json()
    event_type = request.headers.get("X-Github-Event")

    # reviews requested or removed
    if event_type == "pull_request":
        action = payload.get("action")
        if action == "review_requested":
            # TODO: store review request
        return "ok" 
        elif action == "review_request_removed":
            # TODO: delete review request 
        return "ok"        
    return "ok"

    # review submitted
    if event_type == "pull_request_review" and payload.get("action") == "submitted":
    # TODO: update review request
        return "ok"

    # ignore other events
    return "ok"

Github sends different payloads for different events. The event_type is denoted by the header X-Github-Event and the action is denoted by the action field in the payload. The code above just identifies what action triggered the webhook.

As you can see, there are several TODOs in the code. For now, we just return ok to github without actually doing anything with the data. We will handle these events properly after we have implemented storing, retrieving, updating and deleting the review requests' information.

The next step is to verify the signature sent by github. The signature is used for integrity and authentication; it verifies that the payload came from github and that the payload has not been modified by anybody in between.

Create a file called utils.py and add the following code.

import os
import hmac

# get the webhook secret from the environment
WEBHOOK_SECRET = os.getenv("WEBHOOK_SECRET")

# caclulate hmac digest of payload with shared secret token
def calc_signature(payload):
    digest = hmac.new(
        key=WEBHOOK_SECRET.encode("utf-8"), msg=payload, digestmod="sha1"
    ).hexdigest()
    return f"sha1={digest}"

Open main.py and add the code to verify the signature.

from fastapi import FastAPI, Request, HTTPException

import utils

# FastAPI app
app = FastAPI()

@app.post("/webhook_events")
async def webhook_handler(request: Request):
    # verify webhook signature
    raw = await request.body()
    signature = request.headers.get("X-Hub-Signature")
    if signature != utils.calc_signature(raw):
        raise HTTPException(status_code=401, detail="Unauthorized")

    # handle events
    payload = await request.json()
    event_type = request.headers.get("X-Github-Event")

    # reviews requested or removed
    if event_type == "pull_request":
        action = payload.get("action")
        if action == "review_requested":
            # TODO: store review request
        return "ok" 
        elif action == "review_request_removed":
            # TODO: delete review request 
        return "ok"        
    return "ok"

    # review submitted
    if event_type == "pull_request_review" and payload.get("action") == "submitted":
    # TODO: update review request
        return "ok"

    # ignore other events
    return "ok"

Now only github can send you payloads on your webhook endpoint. Let's deploy what we have until now with the deta deploy command. It should take only a few seconds.

$ deta deploy

Implement the review request store

We are using Deta Base to store information about review requests.

We use the python SDK to talk to our database which is pre-installed on a deta micro.

Each item in the database is information about a review request and will have the following schema

{
    "key": str, // randomly_generated
    "reviewer": str, // reviewer
    "pull_request": int, // pull request number 
    "requested_at" : int, // posix timestamp of request
    "submitted_at" : int, // posix timestamp of review submission
    "submitted": bool, // if the review has been submitted
    "crt": int // code review turnaround in seconds
}

Create a file called reviews.py with the following code.

from dateutil.parser import isoparse
from datetime import datetime, timezone
from deta import Deta

# manages storing, fetching and updating review requests information
class ReviewRequestStore:
    def __init__(self):
        # creating a new base (or table) is only one line of code
        self.db = Deta().Base("code_reviews")

    # get review req from pull request number and reviewer
    def __get_review_req(self, pr_num: int, reviewer: str):
        # generator
        review_reqs_gen = next(
            self.db.fetch(
                {"submitted": False, "pull_request": pr_num, "reviewer": reviewer}
            )
        )

        review_reqs = []
        for r in review_reqs_gen:
            review_reqs.append(r)

        # there should be only one corresponding unsubmitted review request
        if len(review_reqs) == 0:
            raise Exception("No corresponding review request found")

        if len(review_reqs) > 1:
            raise Exception(
                "Found multiple imcomplete reviews for same pull request and reviewer"
            )

        return review_reqs[0]

    # store review request
    def store(self, payload: dict):
        # POSIX timestamp
        current_time = int(datetime.now(timezone.utc).timestamp())
        item = {
            "reviewer": payload["requested_reviewer"]["login"],
            "pull_request": payload["pull_request"]["number"],
            "requested_at": current_time,
            "submitted": False,
        }

        self.db.put(item)

    # mark review request complete
    def mark_complete(self, payload: dict):
        submission_time = int(isoparse(payload["review"]["submitted_at"]).timestamp())

        pr_num = payload["pull_request"]["number"]
        reviewer = payload["review"]["user"]["login"]
        review_req = self.__get_review_req(pr_num, reviewer)

        # updates to the review request
        updates = {
            "submitted": True,
            "submitted_at": submission_time,
            "crt": submission_time - review_req["requested_at"],
        }

        self.db.update(updates, review_req["key"])
        return

    # delete review request
    def delete(self, payload: dict):
        pr_num = payload["pull_request"]["number"]
        reviewer = payload["requested_reviewer"]["login"]

        review_req = self.__get_review_req(pr_num, reviewer)
        self.db.delete(review_req["key"])

    # get review requests created since date
    def get(self, created_since: str):
        # posix timestamp
        since = int(isoparse(created_since).timestamp())

    # query submitted reviews created since 'since'
        review_reqs_since_gen = next(
            self.db.fetch({"requested_at?gte": since, "submitted": True})
        )

        review_reqs_since = []
        for req in review_reqs_since_gen:
            review_reqs_since.append(req)

        return review_reqs_since

# initializing a singleton, only one instance should be used
rev_req_store = ReviewRequestStore()

Creating or connecting to the database is only a single line of code if you use Deta Base as you can see in the constructor of the ReviewRequestStore class. It requires no pre-set up of a database.

self.db = Deta().Base("code_reviews")

The ReviewRequestStore class offers methods to store, mark as complete, delete and retrieve the review requests from the database. These methods do the necessary processing of the github payloads to store, update and retrieve only necessary information.

Also, an instance of the class is already instantiated here as it should be a singleton. We will import this instance directly in our main.py and later for the insights.

Now we update our main.py to handle the payloads from github. Open main.py and update the code to the following.

from fastapi import FastAPI, Request, HTTPException

import utils
from reviews import rev_req_store

# FastAPI app
app = FastAPI()

@app.post("/webhook_events")
async def webhook_handler(request: Request):
    # verify webhook signature
    raw = await request.body()
    signature = request.headers.get("X-Hub-Signature")
    if signature != utils.calc_signature(raw):
        raise HTTPException(status_code=401, detail="Unauthorized")

    # handle events
    payload = await request.json()
    event_type = request.headers.get("X-Github-Event")

    # reviews requested or removed
    if event_type == "pull_request":
        action = payload.get("action")
        if action == "review_requested":
            # store the review request
            rev_req_store.store(payload)
        elif action == "review_request_removed":
            # delete the review request
            rev_req_store.delete(payload)        
    return "ok"

    # review submitted
    if event_type == "pull_request_review" and payload.get("action") == "submitted":
    # mark review request complete
        return "ok"

    # ignore other events
    return "ok"

Let's deploy the latest changes.

$ deta deploy

Generate the insights

Now we need to implement retrieving the data from the store and generating the insights with average review turnaround time. We use Bokeh for generating the HTML chart.

Create a file called insights.py with the following code

from datetime import datetime, timedelta
from dateutil.parser import isoparse
from statistics import mean
from math import isnan, nan

from bokeh.plotting import figure
from bokeh.resources import CDN
from bokeh.embed import file_html
from bokeh.models import HoverTool

from reviews import rev_req_store

# manages generating the insights data
class Chart:
    def __init__(self):
        # maps durations to number of days
        self.__durations = {
            "week": 7,  # number of days
            "month": 30,  # number of days
        }

        # maps periods to number of seconds
        self.__periods = {
            "day": 60 * 60 * 24, # number of seconds
            "week": 60 * 60 * 24 * 7, #number of seconds
        }

    # get submitted reviews bucketed by preiods based on duration
    def __get_insights(self, duration: str, period: str):
        if not self.__durations[duration] or not self.__periods[period]:
            raise ValueError("bad duration or period")

        since = self.__get_since(self.__durations[duration])
        submitted_reviews = rev_req_store.get(since)
        return self.__bucket_submissions(since, period, submitted_reviews)

    # convert duration into iso 8601 date format
    def __get_since(self, days: int):
        since = datetime.now() - timedelta(days=days)
        return since.isoformat()

    # bucket submitted reviews based on submission timestamp since date averaged by period
    def __bucket_submissions(self, since: str, period: str, submitted_reviews: list):
        now_posix = int(datetime.now().timestamp())
        since_posix = int(isoparse(since).timestamp())

        buckets = {}
        average_buckets = {}
        separators = []

        # separators are calculated based on period
        # for eg. if period is "day", separators are distanced by 86400 seconds
        start = since_posix + self.__periods[period]
        for start in range(since_posix, now_posix + 1, self.__periods[period]):
            buckets[start] = []
            separators.append(start)

        # fill the buckets
        for rev in submitted_reviews:
            for separator in separators:
                # the separaotrs are sorted in increasing order
                # so a simple comparision suffices here
                if separator > rev["requested_at"]:
                    buckets[separator].append(rev["crt"])
                    break

        # compute average for each bucket
        for separator in buckets:
            date = datetime.fromtimestamp(separator)
            crts = buckets[separator]
            average_buckets[date] = nan  # nan here to denote missing data for the chart
            if len(crts) != 0:
                average_buckets[date] = round(mean(buckets[separator]) / 60, 2)

        return average_buckets

    # generate html chart with bokeh
    def __generate_chart(self, buckets: dict):
        p = figure(
            title="Average code review turnarounds",
            x_axis_type="datetime",
            x_axis_label="date",
            y_axis_label="average turnaround (mins)",
            plot_height=800,
            plot_width=800,
        )
        x = list(buckets.keys())
        y = list(buckets.values())
        p.scatter(x, y, color="red")
        p.line(x, y, color="red", legend_label="moving average code review turnaround")
        return file_html(p, CDN, "Average code review turnarounds")

    # get html chart
    def get_chart(self, duration: str, period: str):
        buckets = self.__get_insights(duration, period)
        return self.__generate_chart(buckets)

    # get json of average values
    def get_json(self, duration: str, period: str):
        buckets = self.__get_insights(duration, period)
        for date in buckets:
            if isnan(buckets[date]):
                buckets[date] = 0
        return buckets

Here we create a class Chart that manages the insights. Chart offers two main methods to get the insights, get_chart and get_json to either get an html chart or a json.

The insights are calculated based on the parameters duration and period.

The main algorithm here is to get the submitted reviews since a specific date, bucket the submitted reviews based on the period and return averages for each period.

# get submitted reviews bucketed by preiods based on duration
def __get_insights(self, duration: str, period: str):
    if not self.__durations[duration] or not self.__periods[period]:
        raise ValueError("bad duration or period")

     since = self.__get_since(self.__durations[duration])
     submitted_reviews = rev_req_store.get(since)
     return self.__bucket_submissions(since, period, submitted_reviews)

# convert duration into iso 8601 date format
def __get_since(self, days: int):
     since = datetime.now() - timedelta(days=days)
     return since.isoformat()

 # bucket submitted reviews based on submission timestamp since date averaged by period
def __bucket_submissions(self, since: str, period: str, submitted_reviews: list):
    now_posix = int(datetime.now().timestamp())
    since_posix = int(isoparse(since).timestamp())

    buckets = {}
    average_buckets = {}
    separators = []

    # separators are calculated based on period
    # for eg. if period is "day", separators are distanced by 86400 seconds
    start = since_posix + self.__periods[period]
    for start in range(since_posix, now_posix + 1, self.__periods[period]):
        buckets[start] = []
        separators.append(start)

    # fill the buckets
    for rev in submitted_reviews:
        for separator in separators:
            # the separaotrs are sorted in increasing order
            # so a simple comparision suffices here
            if separator > rev["requested_at"]:
                buckets[separator].append(rev["crt"])
                break

    # compute average for each bucket
    for separator in buckets:
        date = datetime.fromtimestamp(separator)
        crts = buckets[separator]
        average_buckets[date] = nan  # nan here to denote missing data for the chart
        if len(crts) != 0:
           average_buckets[date] = round(mean(buckets[separator]) / 60, 2)

    return average_buckets

The main algorithm:

calculate the exact time from which we need to retrieve the duration. For eg. if the duration is week, calculate the timestamp of exactly a week ago. This is the since.
get submitted reviews created since the since timestamp
divide up the time between since and now to equal intervals of period by using timestamps as separators. So, besides the first one, each separator will be period seconds higher than the previous separator.
bucket review submissions to the right intervals based on the submitted_at timestamp
calculate average review turnarounds of each bucket

Add api for getting the insights

Now that we have our insights, the final step is to implement the api that the users can get the insights from. For this we offer a GET endpoint to get the insights.

Open main.py and update it to the following code

from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import HTMLResponse

import utils
from reviews import rev_req_store
from insights import Chart

# FastAPI app
app = FastAPI()

# chart
chart = Chart()

## cache generated charts
CACHE_MAX_AGE = 300

@app.post("/webhook_events")
async def webhook_handler(request: Request):
    # verify webhook signature
    raw = await request.body()
    signature = request.headers.get("X-Hub-Signature")
    if signature != utils.calc_signature(raw):
        raise HTTPException(status_code=401, detail="Unauthorized")

    # handle events
    payload = await request.json()
    event_type = request.headers.get("X-Github-Event")

    # reviews requested or removed
    if event_type == "pull_request":
        action = payload.get("action")
        if action == "review_requested":
            rev_req_store.store(payload)
        elif action == "review_request_removed":
            rev_req_store.delete(payload)
        return "ok"

    # review submitted
    if event_type == "pull_request_review" and payload.get("action") == "submitted":
        rev_req_store.mark_complete(payload)
        return "ok"

    # ignore other events
    return "ok"

# get average turnaround insights
# last: for last 'x', 'x' is only one of 'week' or 'month' currently
# period: 'period to calculate average of, currently 'day' or 'week'
# plot: whether to generate a plot or not, returns json if plot is False
@app.get("/turnarounds/")
def get_turnarounds(last: str = "week", period: str = "day", plot: bool = True):
    try:
        if not plot:
            return chart.get_json(last, period)

        html_chart = chart.get_chart(last, period)
        return HTMLResponse(
            content=html_chart, headers={"Cache-Control": f"max-age={CACHE_MAX_AGE}"}
        )
    except ValueError:
        raise HTTPException(status_code=400, detail="Bad duration or period")

We added a route /turnarounds/ to get the insights with three query parameters.

last:str : the duration since the request to get the average turnarounds of, only week or month supported for now, defaults to week
period:str : the period to calculate the average over, only day or week supported for now, defaults to day
plot:bool : whether to view a plot or get a json response, defaults to true

Finally, deploy the changes

$ deta deploy

And we are done. The application should now keep track of the review turnarounds and you can easily get the insights from the api.

If you don't see the application behaving as expected, you can see real-time logs of your application in Deta Visor. To open the visor page, navigate to your micro's visor page on Deta or open it from the cli directly:

$ deta visor open

Deta Base also offers a UI which can be used to easily see what is stored in the database. Here's a screenshot of my base's data with completed submissions.

The entire source code of the application can be viewed on github.

Conclusion

In a matter of few hours we created a github insights tool ourselves (instead of subscribing to an expensive enterprise solution) and deployed it to production effortlessly.

GRT can be easily tweaked and extended to enable additional features:

get individual insights for a reviewer
add other durations and periods
extend the api to accept from and to dates
return min and max turnaround times along with the average for each period
configure the same instance of the app to be used for multiple repositories

Deta enables developers direct their focus primarily on development and implementation of ideas and tools like grt and get them out as quickly as possible.

Building a micro Roam Research Clone from Scratch

Max — Mon, 24 Aug 2020 16:20:45 +0000

Background

At Deta, we believe the individual developer should be empowered to create their own tools in the cloud. We also see that the tools for building these tools are more approachable than ever. What follows is a description of building my own tool, Yarc, to demonstrate this and to scratch an itch within my own workflow.

Roam Research is a notes app which describes itself as a 'tool for networked thought'. Roam aims to free your notes from a rigid hierarchical structure (what they call 'the file cabinet approach') in tools like Evernote. Using Roam, one can easily and deeply network notes using advanced hyperlinking capabilities. For instance, in any given note, one can see all the other notes (backlinks) which link to the said note (bi-directional linking).

I personally liked the bi-directional linking in Roam, but I wanted something lighter weight with the ability to open up the hood and add features as I saw fit, like accessing the raw text of my notes over API. I saw many other permutations of tools that had offered their own take on the bi-directional linking in Roam (see Obsidian, Foam); I named my own clone Yarc (yet another roam clone).

With Yarc, I'm not declaring that this project comes remotely close to matching what the team at Roam has done. Imitation is the sincerest form of flattery, and Roam offers far more advanced capabilities than Yarc will likely every have.

Project Design

What I needed was for the app was simple, consisting of three significant parts:

Core Functionality: a standard for writing notes, uniquely addressing them, and bi-directionally linking them
Backend: the backend needed to serve a note and its contents as well as process updates (content, links, and backlinks) to a given note
Frontend: a client / UI to easily view and update the notes

Core Functionality

First, for the notes themselves, I decided to use Markdown as it is a standard with a syntax versatile enough to support text documents with code snippets, hyper links, pictures, etc, built-in. There is tremendous support for Markdown across many tools, some of which I use in Yarc; if I ever need to migrate any notes out of Yarc, there shouldn't be major conflicts using other tools.

The basic feature in Roam I wanted to emulate was the ability to bi-directionally link notes, triggered by the indicator [[]]. For example, if Note A includes text: bla bla [[Note B]], then Note A should link to Note B, and Note B should have Note A in its list of Backlinks. For this to work, I needed two things:

each note needs a unique address tied to a name
a way to process [[name]] tags as links to that address

The principle which drives Yarc follows from recognizing that Markdown and HTML provide great support for HTTP links out of the box. Consequently, I could give each note a unique HTTP address (e.g: :base_url/notes/:note_name ) and prior to rendering a Markdown note as HTML, pre-process [[note_name]] links by converting them to traditional markdown links [note_name](:base_url/notes/:note_name), while also keeping track of all backlinks in the backend.

To recognize all the unique [[]] links in a markdown note, I wrote a short JavaScript function with a bit of regex that spits out the set of unique [[]] links. When we save a note, we can tell our backend to add the current note as a backlink to all the unique links in the current note.

const getUniqueLinks = rawMD => {
  const uniqueLinks = [...new Set(rawMD.match(/\[\[(.*?)\]]/g))];
  return uniqueLinks;
};

Additionally, we can use the array of current links to create the substitution from a [[]] link to a normal Markdown link ([]()) prior to converting the Markdown to HTML.

This function takes our unique Markdown with [[]] tags and spits out standard Markdown:

const linkSub = (rawMD, links, baseUrl) => {
  let newMD = rawMD;
  for (const each of links) {
      let replacement;
      const bareName = each.substring(2, each.length - 2);
      replacement = `[${bareName}](${baseUrl}notes/${encodeURI(bareName)})`;
      newMD = newMD.split(each).join(replacement);
  }
  return newMD;
};

These two functions form the core of Yarc: recognizing the set of bi-directional links in a note and converting the syntax we use into regular Markdown (which can be converted to HTML). Everything else is tying the database, routes, and UI together.

Backend: Deta + FastAPI + Jinja2

For the backend I used:

Deta to host the compute + api and database
FastAPI as the web framework to do the heavy lifting
Jinja2 to generate the note templates to serve to the client

Database (Deta Base)

I used Deta Base to store permanent data. The database operations are in the note.py file and handle reading and writing the raw note data. The basic data schema for a note is stored under a key (which is the urlsafe version of a note's name), and has the following fields:

    name: str
    content: str
    links: list = []
    backlinks: list = []

Routing and Main Functionality (FastAPI on a Deta Micro)

I used Deta Micros to run the FastAPI application and host it at a unique URL. The routes and business logic for the application are built using FastAPI and are in main.py. FastAPI describes itself as a 'micro framework' and their philosophy of sane defaults and a low learning curve were great in contributing to a quick process of building Yarc. If you know Python, building a web app with FastAPI is a very straightforward process.

There are three primary routes & functionalities in Yarc's backend:

GET /: returns the homepage
GET /notes/{note_name}: returns a note with a given name (and creates the note first if it doesn't exist). Accepts an optional query parameter json=true which returns the note information as JSON.
PUT /{note_name}: receives a note payload, updates the database entry of a given note and all other notes that the note links to (as the backlinks fields need to be updated).

This third route, which keeps track of the correct links & backlinks across notes, was the most involved piece, so I'll include this operation here:

@app.put("/{note_name}")
async def add_note(new_note: Note):
    old_note = get_note(new_note.name) 
    old_links = old_note.links if old_note else []
    removed_links = list_diff(old_links, new_note.links)
    added_links = list_diff(new_note.links, old_links)

    for each in removed_links:
        remove_backlink(each, new_note.name)

    db_update_note(new_note)

    for each in added_links:
        add_backlink_or_create(each, new_note.name)

    return {"message": "success"}

Templating

To serve the notes I used Jinja2 to template HTML files with the note data and the frontend JavaScript code, written in Hyperapp. By injecting the frontend JavaScript into the template instead of importing it as a module, I saved one API call on every page load.

Libraries Used:

Frontend: Hyperapp

For the client side of the web app I used (and learned) Hyperapp. Hyperapp is a super lightweight (1kb, no build step!) framework for building interactive applications in a functional, declarative manner. Having experience using React (+ Redux) where a Component combines state management, a description of the DOM, and side effects, I'd say Hyperapp more clearly de-lineates their concepts (views, actions, effects, and subscriptions). As with other frameworks, there's a bit of a learning to get familiar with their concepts, but once you get a handle on them, it is a joy to work with. Like FastAPI, it lives up to its name, and you can build and ship useful applications super quickly.

For interacting with a note (code in note.js), the Hyperapp application has two primary 'modes' as an end user (toggled by triggering a Hyperapp action):

Edit Mode: This mode displays the raw markdown of a note, allowing the user to write notes
View Mode: This mode displays the note as HTML, allowing the user to follow links

Edit Mode

Editing mode is triggered when the user clicks the edit button, which dispatches the Edit action in Hyperapp. This action also triggers an effect, attachCodeJar, which attaches the text editor I used, CodeJar, to the correct DOM element and binds another action, UpdateContent, to the text editor such that the current state of the text editor is saved into the state tree in Hyperapp.

// Edit Action
const Edit = state => {
  const newState = {
    ...state,
    view: "EDIT"
  };
  return [newState,
    [attachCodeJar, { state: newState, UpdateContent }]
  ];
};

// attachCodeJar Effect
const attachCodeJar = (dispatch, options) => {
  requestAnimationFrame(() => {
    var container = document.getElementById("container");
    container.classList.add("markdown");

    const highlight = editor => {
        editor.textContent = editor.textContent;
        hljs.highlightBlock(editor);
    };

    jar = CodeJar(container, highlight);
    jar.updateCode(options.state.note.content);
    jar.onUpdate(code =>
      dispatch(options.UpdateContent(options.state, code))
    );

  });
};

// UpdateContent Action
const UpdateContent = (state, newContent) => {
  return {
    ...state,
    note: {
      ...state.note,
      content: newContent
    }
  };
};

View Mode

View Mode is triggered by clicking the save button, which dispatches the Save action in Hyperapp and triggers two effects: attachMarkdown and updateDatabase.

attachMarkdown removes the text editor from the DOM element and replaces it with the HTML output from converting the latest note Markdown in state using Showdown.
updateDatabase sends the latest note Markdown, links, and backlinks to the backend to save in the database, via an API call.

Libraries Used for the Frontend

Summary

The full source code of the project is here and includes other bits of the application like the homepage, searching, and interacting with the notes over CLI. It also provides instructions for deployment on Deta if you want to deploy your own instance of Yarc.

There are a number of great tools out there that let you build your own cloud tools remarkably quickly, with little overhead. At Deta, we try to provide no-fuss infrastructure to get your code running. I personally found both FastAPI (for the server) and Hyperapp (for the client) to be really complementary frameworks for building lightweight, personal apps; both are great, no-fuss, options that provide a lighting quick route to getting something out.

Getting Started with Twitter APIs using Deta

Sreeram Venkitesh — Wed, 29 Jul 2020 11:47:40 +0000

The video tutorial to this post

You must have seen Tom Scott's YouTube video where he uses the YouTube API to automatically update the title of the video. I had written a tutorial for the same in a previous article (Check it out here)

After doing the YouTube project, I was thinking if I could do the same with my Twitter account. I tried reading through the Twitter API docs and it turned out doing this was actually pretty easy! I was able to completely write and deploy the code in an hour.

In this post, I'll walk you through how I did it and how you can easily get started with the Twitter API to automate processes like updating your profile, posting tweets etc.

Creating an app with Deta

Create an empty folder for your project
From the folder, run deta new --node twitter-update to create a new Deta Micro.
Once the app is created, you will get some details including the endpoint web address. We will use this to create an app from the Twitter Developer Console.

Setting up the Twitter App

Log in to your Twitter Developer account and navigate to your dashboard.
Create a new app and give it a suitable name. Use the endpoint that you got when creating the Micro as the website url for your app
Go to the keys and tokens tab to find your API Key and API Secret Key.
Under the API keys, click the option to generate your access token and secret. Copy and store these keys somewhere safe as you will only get to see it once.

Before writing the code

Now that we have the necessary keys, lets get to writing the code that actually does the work. But we need to import the API keys and access tokens inside our code inorder to use it.

Create a .env file inside your project and copy the credentials you got from the Twitter App Dashboard into it, as shown

consumer_key=blah
consumer_secret=blahblah
access_token_key=blahblahblah
access_token_secret=blahblahblahblah

Now that you have a .env file, you can access these from inside your code (You might have to install the dotenv package and set require('dotenv').config() inside your js file. I assume you've done this already.)

Writing the code

We are going to use the Twitter for Node.js package for making requests to the API.
Install the package with npm install twitter
Import the twitter package and create a client object

const twitter = require('twitter')

const client = new twitter({
    consumer_key: process.env.consumer_key,
    consumer_secret: process.env.consumer_secret,
    access_token_key: process.env.access_token_key ,
    access_token_secret: process.env.access_token_secret
})

Now you can call the get and post functions of the client object to use the Twitter APIs however you want.

You can refer to the docs to choose an api and study about its parameters and such.

In our example, we need to read the profile details from our account - the name, the number of followers and so on and use it to update a part in the profile, the name.

Let's see how to read the profile first. We can use account/verify_credentials to fetch the details from our Twitter account.

client.get('account/verify_credentials',(err,res)=>{
  if(!err){
    const followerCount = res.followers_count
    console.log(followerCount)            
  }
})

Here we retrieve only the followers_count from the response, but you can try printing the entire response object to see what all information we are fetching.

Now we need to update the profile details. For this we can use account/update_profile.

client.post('account/update_profile',{name: 'New Name'}, (err,res)=>{
  if(err) throw error
  console.log('Updated')
})

This piece of code above will update the name parameter in your profile to whatever value you give it. You can even create a params object with all the stuff you want to change and pass it to the function.

const params = {name : 'New Name'}

client.post('account/update_profile', params, (err,res)=>{
  if(err) throw error
  console.log('Name has been updated')
})

Now that you have the two pieces of the puzzle, you can use the updation inside the function where you fetch the data.

client.get('account/verify_credentials',(err,res)=>{
  if(!err){
    const cnt = res.followers_count

    const params = {name : 'Sree has ${cnt} followers!'}

    client.post('account/update_profile',params,(err,res)=>{
      if(err) throw error
      console.log('Name has been updated')
    })            
  }
})

Now if you run npm start and start the server locally, your Twitter profile name will be updated. But we need this to happen in the background, in real time. This is where Deta Cron comes in.

A cron is a piece of code that has been scheduled to run at a certain interval, and it keeps running again and again. With Deta you can easily deploy a cron with just a few lines of code and a few commands.

For setting a cron with Deta, you can import App from Deta as

const { app } = require('deta')

And set the function to run as the cron inside app.lib.cron like this

app.lib.cron(event => {
    // The code here will be executed repeatedly      
});

So you can bundle up all the above code we used to call the API into a function and call the function from inside here.

Now from your project directory, run deta update -e .env to update your environment variables.
Run deta deploy to deploy your code to production.
Run deta cron set '1 minute' to set the code to run every minute.

Checkout Deta, the cloud for the next billion ideas

Get the code to this project at my GitHub

See the name updation effect live on my Twitter profile

How to use the YouTube API to create a self-updating video title based on views

Sreeram Venkitesh — Tue, 21 Jul 2020 10:31:56 +0000

Link to my video tutorial

You might have seen Tom Scott's video where he uses the YouTube API to automatically update the title of his video in real time to say 'This video has n views'. Sometime after, another YouTuber, Mr.Beast did the same thing with the thumbnail of his video to show how much money he donated.

One of my favourite dev channels, Fireship had posted a video explaining how these people are doing this, explaining how APIs work and how they can be used to automate such tasks.

Although Tom Scott never went into the intricacies of the code behind this process, Fireship explained how to acheive this to a fair degree of detail.

Basically we need to use the YouTube Data API provided by Google to fetch the details of a video. We have two ways to do this

Use an API key to do read-only processes (Like retrieving the number of views)
Use an OAuth 2.0 credential to sign in to your YouTube account and be able to retrieve and update the data.

We would need to use OAuth since we need to update the title of our video.

Fireship's video, although really well made and interesting to watch, is not a tutorial. You can definitely get a head start but not a walkthrough. But I decided to give it a try nonetheless in the hope to being able to document the process and create a more concise tutorial.

Step 1 - Using the YouTube Data API

The API that we're gonna use is has an extensive documentation along with an in-browser API Explorer that you can use to test an API before actually using it in your project.

This can be really handy to test the API while writing the code. Notice how there are two options for selecting the credentials. If you checked Google OAuth and select a scope within it (different scopes have different access levels), you'll be prompted to login to your Google account to authenticate the API request. This way the API can make sure that you are requesting to edit the details of a video posted from your account which you have the access to update.

For the title we are going to need the list and the update functions of the API, both of which you can test from the explorer.

Step 2 - Creating a project with Deta

Being able to call the API and change the video title is not enough. You should be able to do this periodically as the views get updated in real time.

Using Deta makes setting up all of this really easy with their Micros and Cron. You can write a piece of code, deploy it, set a cron and it will keep running periodically after a time period that you specify.

Using Deta Micros and Crons to deploy your code

Create a new Micro

Inside your project directory, run

deta new --node my-project-name

This will create a new directory with the name of your project and it will have a .deta folder inside, with all the information regarding your micro. (If you haven't used Deta before, you'll need to install the CLI and login from your machine. Check out a tutorial here). Once the project is created, you can npm init inside the directory to initialise a node project.

Install the node package for using Google APIs

From your project directory, once you've initialised the node project, run

npm install googleapis

Using the APIs

Once you have the package installed, you can easily access all of Google's APIs for say, YouTube or Drive or whatnot.

const google = require('googleapis')

const youtube = google.youtube('v3')

Here v3 represents DataAPI v3 of YouTube.

You can now use the list function to fetch the video details and the update function to update the details of your video.

youtube.videos.list({
    auth: auth,
    id: 'YOUR_VIDEO_ID',
    part: 'snippet,statistics',
},function(err, result) {
    if (err) {
      console.log('The API returned an error: ' + err);
      return;
    }else{
      //Do something here with the data
      console.log(result.data)
    }
})

OK, so you might be confused with two things I didn't talk about yet.

The auth object we pass to the function are our credentials. This will be the API Key if you only need to read the data or it'll be the access token and refresh token linked to your Google account that has the access to edit the title of the video.

The other arguments for the function are the video ID, which is just the unique string present in the url of every video. The part argument specifies what all types of detail we need to fetch. snippet stands for all the details of the video like the title, categoryId, description, tags etc. statistics is information like the number of views, likes, dislikes and all. Since we need both the title and the view count, we can retrieve both the snippet and statistics of the video.

The Updation

Once you get the details of a video, you can take the view count, create a string saying whatever you want with the view count and send it back to update it as the new title of the video.
Here's how we can do it with code

const video = result.data.items[0]

const { viewCount } = video.statistics

const newTitle = `This video has got ${viewCount} views!`

video.snippet.title = newTitle

youtube.videos.update(
{
    auth: auth,    
    part: "snippet", //since we only need to update the title
    resource: {
       id: 'YOUR_VIDEO_ID',
       snippet : {
          "title": newTitle,   //set title as new title
          "categoryId": 'categoryId'   
       }
},(err, response) => {
    if (err) {
      console.log(`There was an error updating ${err}`)
      return
    }
    if (response.data.items) {
      console.log("Done")
    }
  }
)

You can set the new title from the details collected from the list function and use it with the update function. YouTube also wants us to set the Category ID of the video while updating the details. The complete list of category IDs can be found here

The update function can be called from inside the list function if you want to, to directly access the data in the response of the API.

The Google Developer Console

Now let's look at the auth object that we pass in the function calls. auth is an oauth2client object that has our credentials and information like the access token and refresh token and all.

Remember how when using the API explorer Google asked us to select an account and approve all the permissions? We generate access tokens and store it and pass it to our function calls so that we do not need to do this confirmation every single time the function is called. And since we need to update the title in real time, we will need to call the function frequently.

To generate the access tokens, first create a project in your Google Developer Console.
Add the YouTube Data API v3 to your project from the API library
Go to the credentials tab and create a new OAuth Client ID from the Create Credential menu
You will need to create an OAuth consent screen for this, which you can do easily from the side menu bar in the console.
Once created, download the clien_secret.json file and keep it in yoru project directory. Don't forget to add it to .gitignore if you are going to push your code to a public repository.

There are a few function that we need to run to generate the access token for the first time. This involves running the code from our local machine and confirming the use of the app we created with our Google account.

For doing this, you can clone the repo I made here

After cloning, run npm install inside the directory with the package.json file
Copy the client_secret.json file downloaded from your Google Developer Console into the project directory.
Create a new micro inside the cloned project with deta new. This will create a Micro for you based on the existing files. Once created you can go on and run deta deploy, but we'll wait until everything is done to deploy it.
Run npm start to start the local server on your machine.
You will be given a url in the terminal. Go there and you will get a code in the address bar after you select your Google account and accept the terms. (If it says the app is not trustable, its because we haven't verified it. Don't worry, you are the only person who can access the app.)
Paste this code back into your terminal and this will generate a directory named .credentials in your project folder with your access tokens.
Run

deta deploy

to deploy your micro

deta cron set '5 minutes'

(or however frequent you want the code to run) from the terminal to set the cron

Deta Cron will schedule the running of whatever piece of code you put inside app.lib.cron(event => { });. You can use it to automate whatever routine tasks that you might have.

This blog post was meant to be a tutorial for a use-case for Deta Cron. Hope this gave you some insight into using Deta in your next project!

Check out my GitHub repo for this project here

Making prompts - an app that gets you out of writer's blocks using Reddit APIs and Deta

Sreeram Venkitesh — Sun, 05 Jul 2020 19:21:59 +0000

As developers we know best that sometimes the best way to avoid an idea block is to just start coding (Weeks of coding can save hours of planning, but that's for another story)

Here we are talking about writer's blocks. Coming up with an enjoyable piece of literature can be as challenging as programming, sometimes even more so.

Prompts (live at prompts.deta.dev) is a simple app that I made for writers to exercise their brains hopefully recover from a writer's block.

Prompts works by fetching story prompts from the r/WritingPrompts subreddit and presenting them to you in a minimalistic editor where you can complete the rest of the story.

There are several options to spice up the challenge - you can set yourself a timer and see how much you can write in a fixed amount of time and also calculate your typing speed

Once you are done, you can keep the story you wrote for yourself by exporting it as a document. If you didn't like the prompt you got you can always load another prompt from the menu.

Prompts is hosted with Deta Micros. Deta's Micros or Micro Servers lets you host your Python and Node.js apps in seconds with just their CLI tool. Check out more about them at deta.sh

The code for prompts is open-source at my GitHub here. Any contributions are welcome! (If you are planning to run it locally, the instructions to set up your own Deta Micros is given in the repo's README.md file)

Prompts is also featured in ProductHunt. You can check it out here

What do you think about prompts? What all features would you like to add to prompts? Share your feedback in the comments!

How I built my quarantine project (Thanks to Node.js and Deta Base)

Sreeram Venkitesh — Wed, 01 Jul 2020 09:38:20 +0000

Deta is a cloud platform or a cloud computer as devs call it, and it is built with developer and user experience as the first priority.

Deta Base, the production grade NoSQL database provided by Deta is super easy to use that you can go from nothing to a fully working database for your project in literally minutes!

This post is about how I used Base as the database for one of my side projects and how setting up Base was the easiest part of the project

How does it work?

Once you create account at deta.sh, you'll be taken to your dashboard and will be able to see your projects.

When you are first logging in, you'll have a default project with which you can play around. On opening the default project, you'll get a project key. Copy this and save it somewhere safe because you won't get to see this again!

Once you've got the project key setting up a database is just a few lines of code!

Using Deta in your projects

As of now, you can use Deta Base with you Node.js or Python apps, or as an HTTP API

Based on the stack you are using, you can install the relevant Deta packages -

For Node.js, you can use npm to install Deta

npm install deta

Deta also has a Python SDK that you can get from here

Creating a Base

The app I was working on was a resume builder. The user could login and create a resume of all the things they did while being in quarantine or lockdown during the pandemic.

For this I needed to store the data of the users once they login and their resumes once they create them. I did this by creating two Bases within the same project.

One awesome thing about Deta is that you can have infinite number of 'Bases' or databases within a single project

For achieving this, I referred to the amazing docs and referring to the examples give there.

Steps to create a Base and read/write data

Setting up a Base is as simple as creating a deta object with your project key! You can start writing into and reading from your database right away. Here I'm showing how you can do it with Javascript

In your JavaScript file, first import the package

const Deta = require("deta")

Add your project key to a Deta object

const deta = new Deta("your_project_key")

Now you can create a new database using deta.Base(). Let’s create a sample Base and write some values into it.

const db = deta.Base("users")

Since I need to keep a record of all the users who sign up into my site, I need a users database to store their usernames and email addresses and so on.

I can easily write into the Base once I collect user infotmation from the frontend of my app.

db.put({     
  name: "Sreeram",     
  email: "abc@xyz.com",
  key: "user1"
})

Since Base is a flexible NoSQL database, we can store the data in whatever schema we need or by passing a JSON object we get from the user's input.

db.put({     
  name: "Sreeram Venkitesh",     
  email: "abc@xyz.com",
  socialLinks: [{...}],
  projects: [{...}],
  key: "resume1"
})

One thing to note is that if you do not give a key element manually, Deta will autogenerate a key for the entry. The key can be used to later retrieve the entry from the database.

For my resume project, I used a string made from the email addresses of the users as the key so I can easliy retrieve the users' data from the db once I get the email addresses with which they are signing in the next time (See I told you, setting up Base was the easiest part of this project!)

Reading from the Base

So once I wrote all the code to retrieve the resume from the user and write it into the db, I created a custom link to the user's profile which they can share to show off their resume.

In the backend of this process, I needed to get the user's unique key and search it in Base and retrieve the data stored there.

I could do this easily with the db.get() function

const user_resume = await db.get('user_key');

I can then easily use the returned object to display the data in the user's profile.

quarantineresu.me was the first real Node.js application that I made. I literally had no idea how I could set up a database either. Deta made it really easy for me to get up going with the database so that I could concentrate on other aspects of building my app.

Check out Deta at Twitter or join the wonderful Slack community to meet other developers or for help with coding!