DEV Community: Danny Aziz

How the hell do I use my forked NPM package?

Danny Aziz — Mon, 19 Aug 2019 11:33:44 +0000

You can install your fork by doing npm install github:[GITHUB_USERNAME]/[GITHUB_REPO]

But the package won't work out of the box. Why?

Most of the time the /dist of the package is placed into the .gitignore. So you need to build a packaged version of the package so your project can use it.

To do this there are 2 methods. Only one worked for me.

Method 1 (The one that didn't work for me 🤷‍♀️)

Inside your package.json you add a postinstall that goes into your directly and runs npm install and npm run build

  "scripts": {
    "postinstall": "cd node_modules/[PACKAGE_NAME] && npm install && npm run build"
  },

Now just run npm install and your package should be updated to your fork.

What if it doesn't work?

For a package I was testing it out on, npm install worked perfectly but the build process would never work if the package was already inside node_modules...

Method 2 (Branch method)

This method requires you to make a branch on your fork that will only be used for installing (until the master of your fork gets merged, hopefully)

Create a new branch:
git checkout -b useLocally
Remove /dist from the .gitignore
Add the build command to precommit:

 "precommit": [
     "build"
   ],

Push Branch

git add *
git commit -m "COMMIT_MESSAGE_HERE"
git push origin useLocally

Now install the branch into your project
Just append #[BRANCH_NAME] to the URL of the repo when installing
npm install github:[GITHUB_USERNAME]/[GITHUB_REPO]#[BRANCH_NAME]

Now the /dist will be installed without having to make any changes to the package.json on master!

Serverless Web Scraping With Python, AWS Lambda and Chalice

Danny Aziz — Wed, 24 Jul 2019 12:10:13 +0000

Intro

Programmatic and scalable web scraping is hard to do. There's a lot of build and maintenance involved that has nothing to do with the actual scraping task.

Serverless computing makes it quite a lot easier.

All you need to worry about is scraping the website in a friendly manner.
Let's jump into creating a serverless web scraper with Python and hosting it on AWS Lambda by using Chalice to do all the heavy lifting for us.

We are going to create a small scraper that returns today's #1 product on ProductHunt. (BTW: this is 100% doable using the ProductHunt API)

Here is the repo for the project: https://github.com/DannyAziz/serverless-product-hunt-scraper

1. Set up Dev environment 🛠

Set up your python virtual environment and install chalice

mkivrtualenv serverless_scraping

pip install chalice

Make sure to install this dependency via pip, we will be using this to make our scraping life easier:

pip install requests_html

2. Set up Chalice with AWS 🔑

You need to authenticate your machine with your AWS account. If you've done this before, move on to the next step.

First, you need access keys, to get these: Go to the Security Credentials page from the dropdown in the top right corner

Expand the "Access Keys" dropdown and click "Create New Access Key"

Make sure to note these down somewhere as you won't be able to see these again.

Now you need to save these keys into a AWS config file:

Create the AWS config folder

mkdir ~/.aws

Create and open a new file

nano ~/.aws/config

Paste this in, making sure to replace the keys and region with your own keys and region

[default]
aws_access_key_id=YOUR_ACCESS_KEY_HERE
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
region=YOUR_REGION (such as us-west-2, us-west-1, etc)

3. Create Scraping Script 🕸

Create the chalice project

chalice new-project producthunt-scraper

Inside your chalice project there will be an app.py file, replace the file with the following code:

from chalice import Chalice
from requests_html import HTMLSession

app = Chalice(app_name='producthunt-scraper')


@app.route("/product-hunt/top-product")
def get_top_product_product_hunt():
    session = HTMLSession()
    url = 'https://www.producthunt.com/'
    resp = session.get(url)

    product_list_containers = resp.html.find(".postsList_b2208")

    if len(product_list_containers) == 1:
        product_list = product_list_containers[0]
    else:
        product_list = product_list_containers[1]

    if product_list:
        top_product = product_list.find("li")[0]
        product_obj = {
            "name": top_product.find(".content_31491", first=True).find("h3", first=True).text,
            "url": "https://producthunt.com{url}".format(url=top_product.find("a", first=True).attrs["href"]),
            "description": top_product.find(".content_31491", first=True).find("p", first=True).text,
            "upvote_count": top_product.find(".voteButtonWrap_4c515", first=True).text,

        }
        return product_obj
    else:
        return {"error": "Product List Element Not Found"}

Let's break down the code a little.

All serverless functions in chalice are regular python functions, they need an @app decorator applied to them for the function to be called. In this example, we are using @app.route as we want our function to be called when it's requested via HTTP but you could do @app.schedule to run the function on a schedule or the other methods which you can find out about here: https://github.com/aws/chalice

The main part of our function uses the requests_html package to do all the heavy lifting for parsing the HTML document and pulling out the elements based on class names and HTML tags.
Finally, we either return an object which contains the top product or an error.

4. Deploy 🚀

You can test this scraper out locally via the chalice local command, this will create a local server reachable via http://localhost:8000 that you can test your endpoints with.

Once you're ready to go, use this command: chalice deploy

Chalice will now take care of everything, including creating the AWS lambda function in the console and packaging up all the dependencies for AWS lambda to use.

5. Done 🎉

The deploy command should have spat out a URL, this is the public reachable URL for your serverless function and for our ProductHunt scraper.
You've just created your first serverless scraper!

What is your biggest cause of bugs?

Danny Aziz — Wed, 10 Jul 2019 10:19:24 +0000

Recently I've found a lot of bugs have been caused by my lack of attention to detail.

For example, I was passing in the wrong thing to mixpanel's alias call which caused a month's worth of analytics data to be missing all of the events before a user signs up

Do you think you are causing the bugs in your work? Why?

Deploying React With AWS Amplify Console

Danny Aziz — Tue, 25 Jun 2019 19:05:09 +0000

TLDR:

Setting up AWS Amplify Console/Deploy is really familiar if you've used other CI/CD tools
It doesn't just build your application but also deploys to a managed CloudFront distribution
The build config might confuse you slightly if you're coming from something like CircleCI Intro: AWS Amplify is a new offering from AWS. It consists of two parts, a framework for creating applications that connects your app with AWS services and a CI/CD tool for deploying any application.

Today we will be talking about the deployment service of AWS Amplify.

I stumbled upon AWS Amplify when researching the best way to do cache invalidation on CloudFront distributions.

It turns out that AWS Amplify does instant invalidations on every deploy! (Click here to read more about it)

I'm going to walk you through setting up AWS Amplify with Github and deploying a Client Side Rendered React App.

Let's go 🚀

Breakdown of Steps:

Integrate Github 😸
Connect Repo 🔌
Setup Build Config 🔨
Build Config Gotchas ⚠️
URL Redirects ↩️ Done! ✨ 🎉

Integrate Github 😸

Time Taken: 3 Minutes

Like a lot of CI/CD tools, you connect AWS Amplify with Github (or other version control tools) and on each commit to the repo causes the build process to start.

Connect Repo 🔌

Time Taken: 1 minute

Once you've set up your version control with AWS Amplify, you need to pick your repo and select a branch.

Setup Build Config 🔨

Time Taken: 5 - 10 minutes

This is where the magic begins. If you've used other CI/CD tools than you are familiar with this process.

AWS Amplify will attempt to create a build config automatically, which you can download and place into your repo or edit on the console.

If you have config files for other CI tools, AWS Amplify can also copy these automatically (This is a good and a bad thing)

Build Config Gotchas ⚠️

There are a few gotchas that caused my initial builds to fail. I've gone through the pain, so you don't have to:

sudo Does not work - This is similar to other CI/CD tools
The automatic config might pull things from configs from other CI tools like I said earlier this is a good and a bad thing. I had a lot of commands that were needed for CircleCI to build and deploy but not needed for AWS Amplify - Double check the automatic config.
Build output directory - The warning is there and yet I ignored it. - -Make sure to change baseDirectory to your build processes output folder (Probably /build)

URL Redirects ↩️

If you're running a SPA (Create React App etc), then you will need to set up URL redirects.

In the app project within the console, there will be settings in the left-hand side one of them will be for "Redirects and rewrites"

Here's the basic one for SPA's:

Done! ✨ 🎉

You should be done!

AWS Amplify will deploy your app to CloudFront and give you a URL as well.

Need to do more or having issues? Check out the links below

How do you create static-sites for your dynamic JS projects?

Danny Aziz — Sat, 23 Feb 2019 12:28:02 +0000

So I've got a preact SPA app and I want to create static-sites for some content (The content only updates a few times a year and new content is only added once a week)

Currently, I am using prerenderer-webpack-plugin to prerender all of the routes I require on a build and then using webhooks with Netlify to build on changes.

I'd love to know how other people are approaching this?

Listen to the god damn warnings.

Danny Aziz — Fri, 22 Feb 2019 17:44:41 +0000

(Most) Developers are not silly people, they know what they are doing. So when you (I) are using an open source project and there's a warning that appears, listen to it.

A couple of weeks back I was trying to integrate Elasticsearch into a Django project of mine. Setting up on my local machine was fine as I was using django-elasticsearch-dsl which does all the heavy lifting for you.

Deploying onto my server was a different story...

I had 800,000 objects that I wanted to keep in sync with Django and Elasticsearch, locally this was fine but on my AWS t3.medium server it wouldn't work.

Firstly, the server would become unresponsive when trying to sync - too much RAM usage so I used the queryset_pagination argument to paginate my objects for syncing (Instead of syncing all the objects in one go, it would sync them in bunches).

🎉It's syncing!

30 minutes pass and it's time to check and guess what? There are only around 400,000 objects. Only half of the objects synced! This didn't happen on my local machine which had a copy of the same database, how could this be happening?

Next, I thought Elasticsearch was using up too much RAM and so I separated Django and Elasticsearch onto different servers.

⏳Time to sync again.

500,000 - An extra 100,000 objects but not the 800,000 that I needed. This time I spent more time than I care to admit rummaging through documentation, GitHub issues and source code trying to understand how everything worked.

I tried out different people's forks of django-elasticsearch-dsl but even that didn't help.

and then I saw it...

Pagination may yield inconsistent results with an unordered object_list

This was a warning message that was constantly being displayed as I was syncing but for some reason I ignored it.

Put simply, the queryset_pagination argument which was helping me save RAM (and was required to sync on the server) was the issue. django-elasticsearch-dsl was putting all of the objects into bunches to be synced but there was no order, so some objects were being synced more than once and some were being ignored.

All I had to do was fork the repo and add one line of code: .order_by("id")

Finally, all was good in the world again 🎈