DEV Community: Mandar Vaze

Firefox Reader mode in your Terminal

Mandar Vaze — Wed, 19 Jan 2022 03:52:17 +0000

I'm trying to move to more text based workflows.

On that journey, I came across readable

readable strips out just the readability code from mozilla's firefox engine and provides it as a node module.

npm install -g readability-cli will install a script called readable

By default, readable will strip out all the non-essentials from the webpage and create another HTML page.

One can not use it as-is, cause it has all the HTML tags. So one needs a browser.

Since we are already in terminal why not use w3m ?

Here is how I use it.

readable https://microblog.desipenguin.com/post/fzf-with-fish/ | w3m -T text/html

and here is how it looks 😄

On a related note, w3m official homepage asks one to download tarball.

But your OS's package manager probably has w3m. So I suggest use that instead.

FWIW, I used brew install w3m on macOS.

How to Implement Role based Access Control With FastAPI

Mandar Vaze — Wed, 30 Dec 2020 16:19:08 +0000

What is Role based Access Control (RBAC)

Most of the CRUD apps, require some level of role based access control.

You may have at least two types of users.

Elevated permission user (admin, root or superuser)
Normal user aka everyone else ;)

More likely you have more levels in between.

This means only the users with specific role can access certain API endpoints
or operations e.g. Allow everyone the GET operation, but only admin can DELETE.
Some levels in-between can create/update etc.

Code

Following code assumes your User model has a role attribute.
It is better to have a default value so that every user created starts
with lowest level, even if role is not assigned when creating.

Let us first define the RoleChecker class as follows:

class RoleChecker:
    def __init__(self, allowed_roles: List):
        self.allowed_roles = allowed_roles

    def __call__(self, user: User = Depends(get_current_active_user)):
        if user.role not in self.allowed_roles:
            logger.debug(f"User with role {user.role} not in {self.allowed_roles}")
            raise HTTPException(status_code=403, detail="Operation not permitted")

Then in your routes file use it as follows:

allow_create_resource = RoleChecker(["admin"])

@router.post(
    "/some-resource/",
    response_model=schemas.MyResource,
    status_code=201,
    dependencies=[Depends(allow_create_resource)],
)
def add_resource(resource: schemas.ResourceCreate, db: Session = Depends(get_db)):
    # Some validation like resource does not already exist
    # Create the resource
    pass

Sometimes you want to allow multiple roles to perform certain operation.
That is why, RoleChecker takes a list of roles like :

allow_create_resource = RoleChecker(["admin", "manager"])

Learning (Or how I got here)

If you came here just looking for solution, you can stop reading now.

Read on, to know how I reached the solution, things I tried (and failed)

(Sometimes such details give you an idea for something you may want in the future)

As you may know, you can get the current user details in the API via
Dependency Injection via user: User = Depends(get_current_user)
See the documentation

So easy first attempt was on the lines of

if user.role not 'admin':
    raise HTTPException(status_code=403, detail="Operation not permitted")

I extended the above to user.role not in ["admin", "manager"] to allow
multiple roles to perform that operation.

It works for "proof of concept", but we cant be adding similar code everywhere

Then I created

def verify_role(required_role: List, user: User = Depends(get_current_active_user)):
    if user.role not in required_role:
        raise HTTPException(status_code=403, detail="Operation not permitted")

I needed to pass the list of roles to the function. Unfortunately I could not
call this via Depends. I kept getting Depends has no attribute ... error.

Also, I need to call this from the router decorator function as
dependencies=[Depends(my_func)] rather than in the function param
like user: User = Depends(get_current_user)

Finally another user pointed me to this section of the documentation, and that was that. 🎉

Thanks

I'm grateful for Marcelo aka Kludex and Danny Rohde on FastAPI gitter for the ideas and help.

How to Create Header Images for Your Blog

Mandar Vaze — Thu, 19 Nov 2020 17:07:33 +0000

Option 1 : Quick and dirty Beautiful

Head over to https://coverview.now.sh/ - explore.
It is self explanatory.

When in hurry, this is the best.

It has a nice feature that lets you add an icon based on the technology you are blogging about like python, docker, react, heroku and so on.

While it lets you choose background from bunch of options, you can not (yet) upload your own background image.

Option 2 : More Work, More Control

Create a login on Canva. It is free.
Create New Design 1600x840 - This size works well for most blogs
Go to Background and select something you like
Go to Elements, and select twitter. Choose twitter icon that you like
Go to Text, select a style and then type in your twitter handle.
Go to Text, select a style and then type in your website URL.
Go to Text, select "Add Heading", change font to your liking.
Arrange the text elements such that Heading is prominent.
Optionally, add a little illustration either from Canva or other sites like undraw.co
Optionally, add a technology icon like previous option. (I haven't done that yet)

To avoid doing most of these steps over and over, Save background, twitter and website URL in a template. Each time just change the header (and technology icon, if you have one).

I am trying to use SVG for my blog for two reasons - files are "text" and light. But for some reason, my SVG are only Black and White. I'm sure there are color SVG, but then they are as "heavy" as PNG.

I have found Convertio (link in the Resource section below) to be good option. But there are more options out there. (Including potrace which one can use locally.)

Resources

Why my tests fail only during pre-commit ?

Mandar Vaze — Sat, 05 Sep 2020 16:44:18 +0000

Symptom

Recently I ran across (what I thought was) strange behaviour.

I use pre-commit for all my git commits, and one of the step is to ensure that
all the unit tests pass.

I also have a make target to run just the unit tests.

Each time when I tried commit, the unit test step in the pre-commit
would fail - which indicates that the unit tests did not pass. But when I run
make unit it would pass. 🤷

During the pre-commit, pytest would return with exit code 5 - which
indicates no tests ran. So it wasn't that one or more tests failed during
pre-commit - but nothing was tested - and due to non-zero exit code,
pre-commit prevented the commit (as it should)

Root Cause

It turns out pre-commit runs all the steps only on the modified files.
It passes the list of modified files to each command - implicitly.
That is, unless you configure it not to. 😀

So when I modify a source file (not a test file) - pytest would "ignore"
the file - since it would not contain any tests.

Solution

Now that we understand the "why" - solution is obvious.

Patient : Doctor, it hurts when I do this

Doctor : Then don't do that
😂

For the unit test step, ask pre-commit not to pass the file names as an argument
via pass_filenames: false

Reference

Stackoverflow

Chrome on Docker

Mandar Vaze — Tue, 02 Jun 2020 01:56:01 +0000

Background

Some days ago, I got a call from someone asking for a solution. Due to global pandemic they were unable to hand over a laptop to new employee. They wanted the new employee to access a certain websites but did not wish to share the passwords.

Their idea : provide access to already-logged-in browser to the new person, so they would have access to the sites, but do not need the passwords.

They had considered AWS workspaces - but it is costly. (Finally they got some "Free for a couple of months" deal, and used as their final solution)

I suggested "Chrome in Docker". The conversation ended, but it intrigued me and wanted to try it out myself.

I started with this but it did not work as-is for me. Partially because (I think) the original repo assumes linux host, and I am on macOS. So I had to make some changes.

Summary of my changes

I included the chrome.json in the repo. See details about this file in the Dockerfile comments.
I had to replace libpango with libpangox since the original repo is couple years old.
Removed talk-plugin cause I don't need it.
Created run.sh- because the docker command becomes too long, and changing a param becomes PITA
Removed --device params because it does not apply to macOS (I think)
Finally, I needed XServer, which comes with linux but not macOS. Hence brew cask install xquartz

Bit of Troubleshooting

Where is the fun if things work in the first attempt ?
I kept getting Unable to Open XDisplay error.

To fix the error, I had to set the ip as per this

I created setxhost.sh to simplify this. (Originally it was part of run.sh)

You can see all my changes in my git repo which contains all my
modification.

The README has detailed instructions to try this on your mac.

How to Write Middleware using FastAPI

Mandar Vaze — Tue, 05 May 2020 16:23:46 +0000

For one of my project, I needed to host an API service on the RapidAPI platform. In order to make sure that all the requests to the application are routed via RapidAPI I needed to check special header sent by RapidAPI.

RapidAPI forwards each valid request to the configured server, but injects additional header X-RapidAPI-Proxy-Secret. While a hacker may also send the same header, the value of this header will be only known to RapidAPI platform and your app.

I deployed the server on heroku, and defined an environment variable
PROXY_SECRET which I check against the one sent with the request.

Sometimes I need to test the server directly, in which case I simply do not set this variable (like on my local machine) and this check is bypassed.

Code

import os
from starlette.requests import Request
from starlette.responses import PlainTextResponse

app = FastAPI()

@app.middleware("http")
async def check_rapidAPI_proxy_header(request: Request, call_next):
    # Check if server knows about valid "secret"
    secret_header = os.environ.get("PROXY_SECRET", None)
    if secret_header:
        headers = request.headers
        # If the header is missing, or does not match expected value
        # Reject the request altogether
        if (
            "X-RapidAPI-Proxy-Secret" not in headers
            or headers["X-RapidAPI-Proxy-Secret"] != secret_header
        ):
            return PlainTextResponse(
                "Direct access to the API not allowed", status_code=403
            )

    response = await call_next(request)
    return response

Resources

Uvcorn's ProxyMiddleWare - This is related but was not directly useful for me.
FastAPI middleware documentation
RapidAPI Proxy Secret

How to Process Headers using FastAPI

Mandar Vaze — Sun, 03 May 2020 17:33:49 +0000

This was originally posted on my blog, here.

FastAPI makes processing Headers very easy, just like everything else. There are
two ways to process headers with FastAPI.

Via Request Object

When writing middleware we have direct access to the Request, so it is
much easier to write as :

@app.middleware("http")
async def my_middleware(request: Request, call_next):
        headers = request.headers
        if "X-My-Header" in headers:
            # Do something with X-My-Header

As a Function Param

When writing normal endpoints, it is easier to process headers. One of your
function parameter could be Header itself. FastAPI will do all the work for
you, so that you can focus on the business logic.

See the sample code from FastAPI documentation of this topic.

from fastapi import FastAPI, Header

app = FastAPI()


@app.get("/items/")
async def read_items(*, user_agent: str = Header(None)):
    return {"User-Agent": user_agent}

Important thing to remember that variable name must be in snake_case.

e.g. In the sample code above, we are processing User-Agent header. So
variable name must be user_agent. Names with - are invalid in python. So
FastAPI will convert the dashes/hyphens to underscore for you.

But wait ..

If you really don't want the auto conversion to underscore. just tell FastAPI
that via convert_underscores=False

See documentation below for details.

Resources

Header Param

How to Safely use Google Application Credentials on Heroku

Mandar Vaze — Fri, 01 May 2020 18:06:35 +0000

This was originally posted on my blog, here.

Problem

If you have deployed an app on heroku, you know that the only way to get any files there is via git.
Normally this is fine, but Google API expects an environment variable GOOGLE_APPLICATION_CREDENTIALS that points to a json file.

If you google, you might come across tutorials like this or even SO answer

Essentially they tell you to commit the json file downloaded from Google into git. Since heroku does not have a file system per se, there is no way to get the json file to heroku.

But ..... this file should never be in git. The file contains your
security credentials, and it is (indirectly) linked to your credit card.

Even if hacker does not get (direct) access to your credit card, if they use your credentials, your account will get billed (once you go over quota) and google will charge your credit card.

So putting the credentials json file in git is a bad idea.

So what should a developer to do ?

Solution

The workaround is to store the contents on json file in an environment variable, and at runtime, create the json file on-the-fly by reading the environment variable.

But there is better option, use this Custom Build pack

Even better, Create .profile file in your repo, and create the json file from the environment variable. No buildpack required.

Commands

On your local machine, where you have downloaded the credentials.json
- Store the JSON contents in environment variable : heroku config:set GOOGLE_CREDENTIALS="$(< credentials.json)"
Create a .profile and put this in it : echo ${GOOGLE_CREDENTIALS} > /app/google-credentials.json
- This will create the file on heroku's ephemeral file system each time a dyno is booted.
- Commit this .profile in the root of your git repo
Point to the file as Google API expects.
- heroku config:set GOOGLE_APPLICATION_CREDENTIALS=/app/google-credentials.json
🎉

How to gain experience building large scale apps?

Mandar Vaze — Sun, 07 Apr 2019 17:15:45 +0000

My resume was recently rejected because I do not have experience building "large scale apps" that serve millions of users.

I have over two decades of experience, but mostly in startups/small-ish companies. Thus most of the times I got to work on "new" projects.

Startups always want MVP that can be shipped/released early, so the "growth" is always iterative. Most of the times, the product was discontinued and/or startup closed shop.

While I (believe I) have "theoretical" knowledge about dos and don'ts of building large scale apps, it is not same as having "real" experience from the trenches.

They won't hire me because of lack of "relevant" experience, plus the fact that I am senior engineer. If I were junior experience I assume they would not have this "requirement". I did offer to accept "junior" position (I really liked the company I applied for and their product) but I was told they don't have time for "babysitting" and thus aren't hiring for junior positions at all.

So what do you guys suggest I do to gain "real" experience building large scale app ? (In my case, it is chicken and egg situation)

P.S: I am NOT looking for links to posts like "How to build large scale apps using ...."

P.P.S: If you would like to offer me intership (to gain experience building Large scale app) I'm game. Relevant details available in my profile.

Ethereum Transaction Details for Humans

Mandar Vaze — Mon, 15 Oct 2018 15:50:55 +0000

This was originally posted on my blog, here.

When working on one of my projects, it was asked "How do we ensure that transaction was indeed recorded on the blockchain"

While we can (and did) store the transaction hash returned by the function call in the database, txn_hash alone does no good.

e.g. "Official" method to get the details of the transaction (from the txn_hash) returns something like the following:

web3.eth.getTransaction('0x9fc76417374aa880d4449a1f7f31ec597f00b1f6f3dd2d66f4c9c6c445836d8b§234')
.then(console.log);

> {
    "hash": "0x9fc76417374aa880d4449a1f7f31ec597f00b1f6f3dd2d66f4c9c6c445836d8b",
    "nonce": 2,
    "blockHash": "0xef95f2f1ed3ca60b048b4bf67cde2195961e0bba6f70bcbea9a2c4e133e34b46",
    "blockNumber": 3,
    "transactionIndex": 0,
    "from": "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b",
    "to": "0x6295ee1b4f6dd65047762f924ecd367c17eabf8f",
    "value": '123450000000000000',
    "gas": 314159,
    "gasPrice": '2000000000000',
    "input": "0x57cb2fc4"
}

It doesn't really mean much - even to the developers. Sure from and to addresses provide some useful data, but we don't know what "operation" was performed and with what data. (The data is in the input field shown above, but does not make sense "as-is")

What if you could see something like the following, instead ?

{ name: 'mintToken',
  types: [ 'address', 'uint256' ],
  inputs: [ '5bb4b21e60d0033a1b86b83e4a1f8307ab2d01f9', <BN: 64> ] }

You get the function name, not an address, and the parameters passed to the function - which can be "somewhat" useful, depending on the the types of parameters.

e.g. in the above sample, the first input param is the address and the second one - a number.

It may be easy to guess (from the function name) that we are minting 0x64 i.e. 100 tokens for a given address.

Isn't that much better ?

Getting such "human readable" output is made possible by ethereum-input-data-decoder library - which does the heavy lifting

I added some wrapper functionality to make it as generic as possible without having to make code changes.

You can pass the txn_hash and the path to the JSON file created by truffle compile as arguments to the script.

Thus it works for any contract.

You can see the source code here

If you have any queries (about the code) feel free to file a github issue (or ask here in comments)