DEV Community: Kalob Taulien

Deploying a Wagtail 2.16 website to Heroku

Kalob Taulien — Tue, 08 Feb 2022 19:52:47 +0000

Deploying Python websites is hard. I don't care what anybody says, it's always been the hardest part about Python development in my opinion.

So today I'm going to show you how to deploy your Wagtail website to Heroku fairly easily and for free.

We will be launching a website using:

Wagtail 2.16
Python 3.10
Django 4.0
Heroku

By the end of this tutorial you will have deployed your Wagtail CMS website to the web.

Quickly creating a new Wagtail website on your computer

Ultra fast steps to creating a Wagtail 2.16 website on your computer.

pip install wagtail
wagtail start your_app

This will give you most of the file you need. If you already have a Wagtail project then please ignore the above code snippet 😎

New project files

You will need a couple files for Heroku to read. The first one is the runtime.txt file which tells Heroku which version of Python to use.

runtime.txt 👇

python-3.10.2

Looking for other Python versions? Check them out here.

Then we need a Procfile. This is going to tell Heroku's dynos how to run your website.

Procfile 👇

web: gunicorn your_app.wsgi --log-file -

Note that your_app.wsgi is mapping to the your_app/ folder and the wsgi.py file. If your project is called my_awesome_app, then change:
your_app.wsgi 👉 my_awesome_app.wsgi

Updating your dependencies

We need a few new dependencies for this to work. Open up your requirements.txt file and add these three packages:

requirements.txt 👇

django-toolbelt
psycopg2
whitenoise

Update Wagtail setting files

base.py

In your base.py settings, add the whitenoise middleware below your security middleware, like so:

base.py 👇

MIDDLEWARE = [
    # ...
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
]

Note: You could overwrite your production.py settings instead of using whitenoise in in base.py - any settings that go into base.py are also used in local development, and that's not always what you want to do.

production.py

By default your production.py file is relatively empty. That's because Wagtail doesn't know what tools you'll be using in production, so it makes no assumptions.

We can make some assumptions now that we know we're deploying to Heroku.

production.py 👇

from __future__ import absolute_import, unicode_literals
from .base import *

import os

import dj_database_url

DEBUG = False

env = os.environ.copy()

# It's a GOOD idea to lock this down. 
ALLOWED_HOSTS = ['*'] 

DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

SECRET_KEY = env['SECRET_KEY']

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

COMPRESS_OFFLINE = True
COMPRESS_CSS_FILTERS = [
    'compressor.filters.css_default.CssAbsoluteFilter',
    'compressor.filters.cssmin.CSSMinFilter',
]
COMPRESS_CSS_HASHING_METHOD = 'content'

try:
    from .local import *
except ImportError:
    pass

Working with Heroku

You'll need the Heroku CLI tool. Once you've downloaded and installed the Heroku CLI, open your command line tool and write:

heroku login

Next, you'll need Git. If you don't have Git, make sure you download and install it. Git is the version control system we'll be using to tell Heroku we have new code for our future website. Continue once you have Git installed.

If you are not using git for your project already, initialize a new local repo with (ignore if you are already using git):

git init
git add .
git commit -m "First commit"

That will create a new local repository, add all your files, and create a commit message.

Then create a new Heroku app and connect it to your new git repo with:

heroku create

Then push your committed codebase to Heroku with:

git push heroku master

Now at this point your Wagtail site is in the cloud, but it needs a couple things for the rest of this to work. And this is absolutely vital.

In Heroku, open your app settings and reveal you config vars.

Then add your Django settings module and secret key.

The DJANGO_SETTINGS_MODULE should point to your production settings. In the image above you see testapp.settings.production. That would map to testapp/settings/production.py - your path will certainly be different from mine.

Also you can click here to generate a secret key.

I usually restart all my dynos after this just to make sure these environment variables become available to the website.

Lastly, you can run each of these commands one by one:

heroku plugins:install heroku-config
heroku config:push
heroku run python manage.py migrate
heroku run python manage.py createcachetable
heroku run python manage.py createsuperuser
heroku ps:scale web=1

Now your website is available on the internet! Here's the example website I deployed.

Anytime you need to deploy new changes, all you need to do is:

git add yourfile.ext 
git commit -m "Some new commit message"
git push heroku master

Media is not included

Media (files uploaded by the users through the Wagtail admin) are not supported in this tutorial. Heroku doesn't come with file storage and it's best to use something like AWS S3 for that purpose. All your static assets will be served using whitenoise, but your uploaded files won't work - here's a blog post on getting started with S3 on Heroku with Wagtail

Did you find this helpful?

If you found this article helpful please let me know on Twitter @KalobTaulien - I'm always interested in hearing from the Wagtail community.

Top 5 websites to learn just about anything

Kalob Taulien — Mon, 17 Jan 2022 19:03:07 +0000

Top 5 websites to learn just about anything

Learning shouldn't stop after high school or college.

Whether you're looking to change careers or just learn something new to fight your boredom, there are a lot of great online platforms that can help you learn new and exciting subjects.

In this article I'm listing the top 5 learning websites and have broken them down into categories.

Best Selection: Udemy
Best Price: Arbington
Best Celebrity: Masterclass
Best University Courses: Coursera
Best Short Creatives: Skillshare

Udemy

https://www.udemy.com/

Udemy is a super popular learning website that says they have over 150,000 courses to choose from. That's why they've earned the "best selection" title in this article.
Now, I did an audit, and found that over 50,000 classes don't even have one student. So take that 150k with a grain of salt.

But you'll still find tonnes of high end teachers and courses, ranging from Angela Yu to Colte Steele and Kalob Taulien (that's me!).

Classes range in price from $20-$200, but frankly not a single person on the web has bought a course from Udemy for $200 in the last 5 years. Don't be fooled by the higher prices, this is just a marketing tactic to make you feel like you're getting a better deal - it's called price anchoring, and it makes Udemy feel like Groupon.

Udemy will discount their courses down to $10-$15 every couple of weeks so just wait for that deal or lookup coupon codes by Googling "Udemy Coupon"

Pros:

You can learn just about anything on Udemy
It's quite affordable if you just take a handful of courses
30 day money back guarantee on every class
Certificates included

Cons:

Must buy individual classes and their pricing seems like a scheme
Incredibly toxic for the teachers

Arbington

https://arbington.com/

Arbington.com is a new learning platform that has nearly 2,000 classes for a great price (if you subscribe).
They have hundreds of courses ranging from coding and design to watercolor painting, business growth and much more.

The quality from what I could see was mostly great, too, we were pleased to see they have a standard for quality.

By the looks of it plenty of top teachers from other platforms that I recognized have started moving over to Arbington. So that's a good sign.

Are you interested in learning how to code? Arbington.com is the best place for this. $15/month gives you access to over 200 web development courses. If this were Udemy, that would cost over $2,000.

Classes are all priced at $19.97 so you always know what you're going to be paying for, which is a tad higher than Udemy's discounted prices. But there's also a subscription plan for $15/month that gets you access to every course.
Overall, Arbington earned the best price award in this article for having the best price with the best quality and selection.

Pros:

They include a subscription for $15/month to access over 1700 classes with a 14 day free trial
Classes can be individually purchased with a 30 day money back guarantee
It's very affordable
Certificates included

Cons:

You can subscribe OR buy individual classes which feels a bit confusing at first

Skillshare

https://www.skillshare.com/

Skillshare has the best short creative classes. If you're looking for a tonne of painting, drawing or doodling classes this is the platform for you.

They say they have over 27,000 classes however a quick audit showed us that's not the real number, it's closer to 19,000. But still, that's A LOT of content and that's why they've earned the Best Short Creative award in this article. They are well known for their super short courses (as short as 10 minutes) so you can power through a dozen classes in a single day if you wanted to.

They are completely subscription based so you can watch as many classes as you want.

The quality is pretty good too! Makes sense since artistic people love creating high quality content.

However they seem to promote their famous teachers more and more, making them feel more like MasterClass and less like a supportive community of creatives. And they have hundreds of "staff picks" which tells us they favor certain teachers and rank others lower, which feels a bit unfair in a marketplace.

Pros:

Huge selection of courses
Affordable

Cons:

Pricing seems to change based on the country you're in
They'll randomly delete thousands of courses which could include a class that you enrolled in and are enjoying
Support is non-existent
No certificates

Coursera

https://www.coursera.org/

If you are looking for university classes to take online, Coursera is the place for you! It's a bit pricier than other learning websites, but you're also getting top-notch education from university teachers.

There's A LOT to learn, and the classes are all amazing. I've taken a number of classes from Coursera and was never unhappy about it. And was even happy to buy the certificate at the end.

If you're looking for university credits, however, check out Outlier.org - I haven't tried Outlier but they look truly amazing.

Most courses come in the form of a cohort, so you start and end on certain dates (not self-paced). Usually that's a good thing, but the courses I've taken never felt like there was a real community or cohort - just deadlines.

Pros:

Highest quality education on the internet.

Cons:

Less affordable but it's a fair tradeoff for the quality of content you're buying
If you want a certificate you must pay for it

MasterClass

https://www.masterclass.com/

MasterClass is in a category of its own. I call it "edutainment" - short for education/entertainment.

You'll be learning from famous people. From cooking to guitar playing and film making. You can even learn from Bill Clinton now!

The videos you'll be watching are expertly crafted as if HBO were producing them.

What's cool is you get to spend time with someone who is famous, and possibly someone you look up to.

Starting at $20/month (and going as high as $30/month) you get access to quite a few classes. It's not thousands of classes, but you'll still be impressed with their selection and the instructors you get to work with.

Classes are short, though, being roughly 4 hours long. If you're learning something hard, like guitar from Tom Morello, 4 hours won't feel like enough.

Pros:

Learn from people who actually DO it and have made it
Affordable

Cons:

The classes are shorter than expected
No certificates

Depending on what you're looking for, there's a type of learning website for you.

If you're looking to get the best bang for your buck, Arbington.com is the way to go. If you want ultra high quality education then Coursera is definitely the best route. And if you want a hybrid of education and entertainment, MasterClass is probably what you're looking for.

How to create a PyPi (Python) package

Kalob Taulien — Thu, 06 Jan 2022 21:04:03 +0000

Have you ever wanted to create a Python package so you can type the following?

pip install my-awesome-package

If so, this tutorial is for you.

1. Getting Poetry

Poetry is a system package you can use to very easily manage your Python Packages.

It makes handling your packages super easy. I have 9 Python packages under my name, and this is by far the best solution.

First, you need to install Poetry on your system.

osx / linux / bashonwindows install instructions

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

windows powershell install instructions

(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python -

More details about installation here

Now you can type poetry --version to see the version you're running. If this doesn't work for you, try closing and re-opening your terminal.

2. Creating a Poetry project

In this tutorial we're going to create a new package from scratch.

Go ahead and type:

poetry new test-package

As the docs mention, it'll create a new folder with a bunch of files for you that are structured like this:

test-package
├── pyproject.toml
├── README.rst
├── test_package
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_test_package.py

3. Adding your code

Create a new file in test-package/test_package/ called myfile.py.

Your file structure should look like this now:

test-package
├── pyproject.toml
├── README.rst
├── test_package
│   ├── __init__.py
│   └── myfile.py  # <- This is new
└── tests
    ├── __init__.py
    └── test_test_package.py

All of your code will go in the test_package/ folder.

Note: This folder name will be different if you used a different package name during step 2.

You can add any files you want in this folder. This is what get's bundled in your package.

Now in your myfile.py file, add this function:

def custom_function(name):
    print(f"Hello {name}!")

Later we can use this code like so:

from test_package.myfile import custom_function

👆 We'll get to that later, first we need to build the package.

4. Building your package

So far we just have a bunch of code sitting on our computer and if someone else wants to use our code, they need to copy all the files and folders.

Let's "build" the project.

poetry build

This will take all your code, bundle it, and put it in a versioned file inside of a /dist/ folder. The file should look something like this: test-package-0.1.0.tar.gz.

It will also create a wheel file for you that looks something like this: test_package-0.1.0-py3-none-any.whl

5. Testing your code in other virtual environments

Now what I like to do is test my package in a brand new virtual environment.

Create a new virtual env in a new folder somewhere else on your computer.

mkdir testenv && cd testenv 
python3 -m venv .venv/ 
source .venv/bin/activate

Note: Your virtual env will be created differently if you're on Windows. You can use virtualenv, pipenv, Docker, etc. It just needs to be a fresh environment.

Now that you're in a new Python environment, copy that .tar.gz from from earlier into this folder. (I usually use the operating systems OS for this, but there's nothing wrong with using cp to copy the file). The only file in this virtual environment should be your .tar.gz file.

Now install this file with:

pip install test-package-0.1.0.tar.gz

Assuming everything went well, let's test this out.

Open a new Python shell (REPL) with:
python (on non-Windows) or py on Windows. And type this out:

from test_package.myfile import custom_function

custom_function("Kalob")

It should print out Hello Kalob!.

And just like that, you have create a bundled package. But it's not available for everybody just yet.

6. Push it to GitHub

At this stage I like to push my code up to GitHub.

Because we're making a public package, I tend to make my repo's public as well. No use hiding the source code when it will be easily downloaded from PyPi.org anyway.

Dont forget to add /dist/ and other files to your .gitignore file. The /dist/ folder is where your bundled code goes when it's built by Poetry.

7. Publish the package

Now let's publish this code on PyPi.org.

First, you need a free PyPi.org account. You'll need your username and password handy because Poetry will ask for it so it can upload your file to PyPi and associate it with your account.

Once you have a free PyPi account, go ahead and run this in your terminal:

poetry publish

It will ask for your username and password. Go ahead and type those in.

And now your package will be on PyPi.org.

Note: If the package of test-package already exists, or at one point existed, on PyPi.org you cannot use the same name. Making your Python packages have unique names is the best way to avoid this problem.

If you didn't run into a naming problem, you'll have successfully published a package to PyPi.org!

Congrats!

8. Install your package using pip

Assuming your package is now on PyPi, let's install your package in another project and test it out.

pip install test-package

Everything should work as expected now. And your code is now easily accessible to every Python developer. 🎉

Example repo

If you'd like to see a simple package I recently created for Arbington.com take a look at python-vimeo on GitHub and feel free to use any part of that repo. That's why we open sourced it! :)

What type of developer would a startup hire?

Kalob Taulien — Tue, 04 Jan 2022 16:39:53 +0000

"The world has a shortage of developers."

You might have heard this before. It's fairly common for people to say this.

It's kind of true, but not really...

Here's the truth.

"The world has a shortage of [senior] developers."

Because companies don't like hiring junior or intermediate developers because they cost more to get up to speed.

And the average developer only stays at a job for 2 years.

But also... senior developers are expensive.

At Arbington.com we would love to hire a senior developer, but it'd also cost us like $120,000-$150,000 for the best possible candidate.

And that's not an option at a lot of startups.

So they hire intermediate developers, if possible. And if that's not an option, then they hire juniors.

And they'll label the junior job as an "internship".

Don't be fooled, interns need to make money too. I would never EVER advise anybody to work for free. But minimum wage if it's your first job? Possibly.

So this begs the question...

What are startups looking for in a developer?

Skills, of course. Can you code? Great! Are you the worlds best coder? It doesn't matter if you are or if you aren't.

Not sure what stack they use? Just ask! Tech stacks aren't secrets.

But fullstack is key because a full stack developer is basically a backend dev and a frontend dev at the same time. They aren't usually AMAZING at either, but are super flexible and dangerous with what they're given.

Leadership attributes. When joining a startup, you're not just a developer - you're a core part of the team! You help develop the future culture of the company, even if you are employee #20.

Adaptability. Startups move FAST. You need to be able to adjust. If your dream is to work for the government, a startup is absolutely not for you.

Flexible pay. Startups are cash-poor, usually. Unless they just raised a few million dollars you're likely looking at a lower pay rate. But you can get stock options - so if you're really in love with what they do as a business, and you think it's going somewhere, stock options are a great way to support the company and get paid at the same time.

A curious personality is also vital. Be curious about how to help the startup. Be curious about how to level up your code. Be curious about life in general! Because your curiosity is going to help develop the company culture, and drive you to become an insanely amazing developer.

Do you need to be the best? No!

But you do need to have a lot of the soft skills and desire to become better at the technical portions.

How do you get in?

I've mentioned this before in a Twitter thread I wrote, but I'll say it again:

Resumes are dead. Friends hire friends.

Get to know the founders. Email them, support them, offer them help.. earn your way into their circles.

Actually, this is just good advice for any job.

But once they get to know you a little bit they'll be more likely to hire you.

Why?

Because you aren't seen as a "risk" anymore, but rather you'll be seen as a friend to the company - and a GREAT candidate to hire.

I hope this was helpful!

If you found any value in this article I would love if you could share it or follow me here on Dev.to.

How to create a Flask app in Python

Kalob Taulien — Fri, 31 Dec 2021 21:32:04 +0000

Let's create a Flask app from scratch, using Python 3.9 or newer.

Note: We are not deploying the app, we're just getting started. If you want to deploy a Flask app to Heroku, check out this article.

We'll also get started with a basic templating system. Nothing drastic, but something to add custom information to your templates.

Step 1: Create a virtual environment

There are LOTS of ways to do this, but the simplest way is to use a venv.

mkdir yourproject
cd yourproject/
python -m venv .venv/ 
source .venv/bin/activate/

Step 2: Install Flask

Once you are inside your virtual env (the last line from the code above will get you "inside" your venv), you can simply run:

pip install flask
pip freeze > requirements.txt

This will install Flask in your virtual environment, and create a requirements.txt file for other people to use when installing your project.

Step 3: Your main file

Your main file is where your initial code will live. Go ahead and create main.py and add this to it:

from flask import Flask
app = Flask(__name__)


@app.route('/')
def index():
  return "<h1>Hello world</h1>"

Now, this won't actually do anything yet. We need to run this code. We'll do that in the next step.

Step 4: Environment variables and running the app

First, let's set a couple environment variables. This will help tell Flask what to do while we're developing our app.

In your command line (while inside of your virtual environment) run:

export FLASK_ENV=development
export FLASK_APP=main

Next, we can run:

flask run

And you should see something like this in your terminal:

 * Serving Flask app 'main' (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 814-739-543

Head on over to http://127.0.0.1:5000/ in a new tab in your browser, and you should see Hello World in big bold letters.

Step 5: Adding a template

Templates make life easier. You can write a basic .html file with some Jinja templating in it for dynamic content. But let's start with a simple template.

Open your main.py file and change it to look like this:

# main.py 
from flask import Flask, render_template
app = Flask(__name__, template_folder='templates')


@app.route('/')
def index():
  return render_template('home.html')

This tells Flask to use the templates/ folder as the main folder for templates, and to use home.html as the main page. You can rename home.html to anything you like.

Note: There's A LOT to know when it comes to templating in Jinja, but we aren't going to cover very much right now so we can keep this simple.

Now create a new folder beside your main.py file called templates/ and add a child file called home.html.

Your file structure should look like this now:

And your home.html file should have this in it:

<h3>Hello from home.html</h3>

Step 6: Creating a second view.

In your main.py file, let's create a second "view". This is the route (sometimes called path) to another page.

This time we'll add something dynamic to the template.

At the top of main.py add this import:

import datetime  # <- Add this line 
from flask import Flask, render_template
....

And at the bottom for main.py add these lines:

@app.route('/time')
def timetest():
    now = datetime.datetime.now()
    return render_template('time.html', now=now)

Lastly, add templates/time.html to your project, and inside of time.html add:

<h3>The time is {{ now }}</h3>

Now navigate to http://127.0.0.1:5000/time and it should show your time. And it will update every time the page is refreshed.

Note: If the time doesn't match your exact time right now, it's because your Flask server thinks it's on a different timezone.

What we did was:

Create a variable called now with the current date/time.
Added , now=now to the render_template() function. This adds "context" to your template.
In your template, you can access {{ now }} as a dynamic value.

Your project should look like this:

Your project so far should be structured like this:

And all the files I wrote can be found on GitHub here: https://github.com/KalobTaulien/flask-tutorial

Building a scraping tool with Python and storing it in Airtable (with real code)

Kalob Taulien — Thu, 30 Dec 2021 00:27:30 +0000

A startup often needs extremely custom tools to achieve its goals.

At Arbington.com we've had to build scraping tools, data analytics tools, and custom email functions.

None of this required a database. We used files as our "database" but mostly we used Airtable.

Scrapers

Nobody wants to admit it, but scraping is pretty important for gathering huge amounts of useful data.

It's frowned upon, but frankly, everyone does it. Whether they use an automated tool, or manually sift through thousands of websites to collect email addresses - most organizations do it.

In fact, scraping is what made the worlds best search engine: Google.

And in Python, this is REALLY easy.

The hardest part is reading through various forms of HTML, but even then, we have a tool for that. Let's take a look at an example that I've adjusted so you can scrape my website.

We'll use https://kalob.io/teaching/ as the example and get all the courses I teach.

First, we look for a pattern in the DOM. Open up that page, right click, inspect element, and look for all the blue buttons.

You'll see they all have class="btn btn-primary". Interesting, we've found a pattern. Great! We can work with that.

Now let's just right into the code. And if you're a Python dev, feel free to paste this into your terminal.

import requests 

response = requests.get("https://kalob.io/teaching/")
print(response.content)

You'll see the HTML for my website. Now, all we need to do is parse the HTML.

Note: utf-8 encoding is most commonly used on the internet. So we'll want to decode the HTML we scraped into utf-8 compatible text (in a giant string)

Our code now looks like this:

import requests 

response = requests.get("https://kalob.io/teaching/")

html = response.content.decode("utf-8")
print(html)

And you'll see the HTML looks a little nicer now.

Now here's a big hairy problem: parsing HTML. Some people use attr="" some people use attr='' some people use XHTML and some don't.

So how do we get around this?

Introducing: Beautiful Soup 4.

In your Python environment pip install this package:

pip install beautifulsoup4

And your code now looks like this:

import requests 

response = requests.get("https://kalob.io/teaching/")

html = response.content.decode("utf-8")

import bs4  # You'll need to `pip install `
soup = bs4.BeautifulSoup(html, "html.parser")
print(soup)  # Shows the parsed HTML
print(type(soup))  # Returns <class 'bs4.BeautifulSoup'>

So our soup variable is no longer a string, but an object. This means we can use object methods on it - like looking for certain elements in the HTML we scraped.

Let's put together a list of all the links on this page.

import requests 

response = requests.get("https://kalob.io/teaching/")

html = response.content.decode("utf-8")

import bs4  # You'll need to `pip install `
soup = bs4.BeautifulSoup(html, "html.parser")

courses = soup.findAll("a", {"class": ["btn btn-primary"]})
print(courses)

Look at that.. now we have a list of buttons from the page we scraped at the beginning of this article.

Lastly, let's loop through them to get the button text and the link:

for course in courses:
    print(course.get("href"))
    print(course.text.strip())
    print("\n")

Listen, I wrote 3 print statements to make this clear - but typically I'd write this in a single line.

Now we have something to work with! We have the entire HTML element, the href attribute, and the innerText without any whitespace.

The entire script is 9 lines of code and looks like this:

import requests 
import bs4  # You'll need to `pip install `

response = requests.get("https://kalob.io/teaching/")
html = response.content.decode("utf-8")
soup = bs4.BeautifulSoup(html, "html.parser")
courses = soup.findAll("a", {"class": ["btn btn-primary"]})

for course in courses:
    print(f"{course.get('href')} -> {course.text.strip()}")

Moving this data somewhere useful.

You know me, I'm a HUGE fan of Airtable.

And instead of using local database or a cloud based database, I like to use Airtable so me and my team and work with the data and easily expand the tables if we need to. Like if we needed to add a column to see if a course meetings our criteria to be on Arbington.com.

For this we use Airtables API and the python package known as
airtable-python-wrapper.

Go ahead an install this through pip.

pip install airtable-python-wrapper

Now before we continue, you'll need a free Airtable account 👈 that's our referral link. No need to use it, it's just a nice kickback for us for constantly promoting Airtable 😂

Once you have an account, you need to dig up your app API key, your table API key, and your Base Name. It would look something like this in python:

from airtable.airtable import Airtable

airtable = Airtable('appXXXXXXXXX', 'Links', 'keyXXXXXXXXXX')

Lastly, all we need to do is create a dictionary of Airtable Column Names, and insert the record.

import requests 
import bs4  # You'll need to `pip install `
from airtable.airtable import Airtable

response = requests.get("https://kalob.io/teaching/")
html = response.content.decode("utf-8")
soup = bs4.BeautifulSoup(html, "html.parser")
courses = soup.findAll("a", {"class": ["btn btn-primary"]})

airtable = Airtable('appXXXXXXXXX', 'Links', 'keyXXXXXXXXXX')

for course in courses:
    new_record = {
        "Link": course.get('href'),
        "Text": course.text.strip(),
    }
    airtable.insert(new_record)

Assuming you setup your Airtable columns, table and API keys properly, you should see my website links and URLs appear in your Airtable.

Now you and your team can scrape webpages and store the data in Airtable for the rest of your team to use!

Pulling data out to work with it

Now that all the data we want is in Airtable, we can use the same Python package to pull the data out, work with it, scrape more data, and update each record.

But that's for another day 😉

Want to learn Python?

If you're looking for online courses, take a look at Arbington.com, there are over 40 Python courses available.

And it comes with a free 14 day trial to access over 1,500 courses immediately! 🔥

What to learn to become a fullstack developer (for beginners)

Kalob Taulien — Tue, 21 Dec 2021 17:11:11 +0000

Learn to earn in web development

In web development there is a lot to learn before you can start to earn.

Getting into web development, or coding/programming of any kind, can take quite a while.

Learning is the easy part, but landing a job… well that's the hard part!

But that wasn't always the case. Once upon a time, it was the exact opposite.

You see, when I first started learning how to code in 1999 all we had were boring books and a few awful websites.

But nowadays you can just watch videos and the information is downloaded directly into your brain, how awesome is that?!

But in 2021/2022 we have a problem… there is SO MUCH information we don't know what to follow or believe.

That's where online courses come to save the day.

Online coding courses give you the knowledge and practice you need to learn how to code, and they give you the guidance you need to move forward as a web developer looking for a career change.

What do you need to learn first (in web development)?

HTML.

It's easy. You could learn and be REALLY good at HTML in less than a week.

Every website uses HTML, so this is the first thing you'll want to learn.

Then you need to learn CSS.

CSS is also easy, but it's BIG and has strict rules that aren't always obvious. It should take you about a month to get really good at CSS and to start making beautiful and modern looking websites.

Then you need to learn JavaScript.

Ugh, JavaScript.

I say "ugh" because JavaScript is a HUGE ecosystem full of shiny objects that don't matter. Don't learn React first, don't learn Vue first, don't learn jQuery first.. learn plain, vanilla, raw, unfiltered, boring JavaScript first.

Don't get distracted by all the tools, packages and fun things you can do. Just keep it simple. This can take you between 1–3 months to learn.

JavaScript is a programming language, whereas CSS and HTML are not programming languages - they are considered markup languages. And the human brain registers a programming language as a different way to think - it's literally like learning a new speaking language. So be patient at this step.

Then you need to learn about APIs.

Application Programming Interfaces.

That sounds harder than it actually is. It's honestly just fancy talk for "sending and receiving data from a different computer over the internet". It's like asking your waiter for a drink - you make a request, it shows up, and what you do afterwards is up to you.

API's should take you about a day to learn the theory. But then another 1–2 weeks for practice.

Next up, learn Git & GitHub

Git is a "version control tool". It lets you make changes over time, rewind your work, save it and store it somewhere safe like GitHub.com or GitLab.com.

It can get REALLY complex really easily, so just learn the basics for now.

This should take you about a week to learn, but then you should be using Git on every project moving forward for all of time. So you'll get REALLY good at it pretty quickly.

Lastly, learn a server-side language

Server-side programming languages are the languages you write that exist on the server. That could be Python, PHP, Java, JavaScript, etc.

I personally like Python because it's powerful, easy to learn, easy to read and write, and it's the worlds #1 most popular programming language - so you know there are jobs related to Python.

This is similar to learning JavaScript, time-wise at least. It should take 1–3 months to learn a server-side language well enough you can apply for a job.

If you already know JavaScript quite well, you'll pick up other languages quite easily. So if you already know JavaScript and you're just learning Python, it'll probably take half the time to learn Python, so maybe 2–6 weeks.

Where can you learn all this?

Arbington.com is the place to learn this. Right now it's $15/month to get access to over 150 web development courses, plus another 150 design courses, hundreds of marketing, SEO and analytics courses, and so much more.

In total you'll get immediate access to over 1,500 courses for just $15/month - and it comes with a free 14 day trial.

Certificates of completion are included in every course!

And just like that…

In about 7–10 months you have become a junior fullstack web developer.

There are a few things we skipped over like learning React.js or Django for Python, but once you know the programming language the frameworks become easier to learn.

Don't forget to open source all of your work on GitHub to future employers can see your progress!

Happy coding!

Tools I use everyday as a senior web developer 🔧

Kalob Taulien — Mon, 20 Dec 2021 19:20:03 +0000

Hey everyone!

I'm going to go off track in this article and talk about the tools I use on a regular basis. These are tools or languages that I need as a startup and as a senior web developer.

At Arbington.com we use all of these tools, too.

Let's start with the ones you're most likely interested in: developer tools!

Developer tools

Every day I work on the code base at my startup. And every day I use these tools.

This isn't the full list, I'm sure I'm forgetting tools and languages, but these are the immediate ones that come to mind as a full stack web developer.

Languages

Everyday I write JavaScript and Python.

JavaScript, because, well, what's the alternative for client-side coding? Not much, really. So I'm stuck with it and make the best of it. 🤷‍♂️

I also use JavaScript in terms of Webpack to compile my JavaScript and CSS. That's server-side JavaScript through Node.js, which isn't exactly the same as browser-based JavaScript. It looks the same, but has a different purpose.

And Python, because, well, it's powerful, fast, easy to maintain, and the ecosystem has every possible package I could think of.

Tools

The command line! Holy smokes is this important. Don't discount it as "and old way of doing things". Modern devs ALL use the command line for everything from running tests, compiling code, accessing servers, and committing code to GitHub.

Speaking of GitHub, Git and GitHub. I use GitHub to host my repositories, but you can just as easily use BitBucket or GitLab. And I strictly use Git on the command line because that's what servers use - so if I ever need to SSH into a server, I'm well equipped with the knowledge and tools I need to pull down my latest changes.

Docker is another big one. I have multiple projects on my computer that user different versions of Python or Node.js.

Instead of installing all these different versions (ie. Python 3.6, 3.7, 3.8 and 3.9) I simply use a Docker container that downloads that version and compartmentalizes it in a virtual machine-like instance. If I no longer need it, I just delete it. And it has no effect on my laptop.

Developer tools, in the scenario of a browser's dev tools. Namely, I use Chrome and Firefox. But all major browsers have dev tools. It's great for debugging JavaScript and Ajax/Fetch requests to see what your server is returning.

VS Code is the editor I choose to use. I was late to the game because I used to love Sublime, and Notepad++ before that (is my age showing, yet?). But I'm really happy with VS Code and it has built in support for JavaScript, TypeScript and Python with extensions for a Git GUI, Docker container management, and more. I still prefer the command line for most things, but not everyone is as command-line-focused as I am.

That's my day-in-day-out set of tools. Pretty simple, right? You don't need much to do a lot these days.

I also make use of cURL, Vim, ipython, Jupyter notebooks, tmux, Docker compose, the kubernetes command line tool, and plenty of other fun tools. But if I didn't have any of these on a typical day, I'd be OK to carry on.

Non-dev tools

I use quite a few non-dev tools in my day to day life. And you probably use a lot of these too.

G Suite or whatever it's called these days - we use this for hosting email and creating internal documents at Arbington.com.

Calendly is new to my tool belt, and I'm happily paying for it. It hooks into numerous calendars and lets me book times with teachers and students, and send follow up emails.

Stripe.com for processing payments. I have an article on how to get started with Stripe to accept money from customers through your website. I also use PayPal to pay my vendors, staff, teachers and other misc. expenses.

Airtable is a service I would be lost without. It's like Google Sheets, but on steroids! It's powerful, fast, has a great API, and lets me put our teams data into a centralized place to digest, keep track of tasks and clients, and has version history on each record.

Dropbox for storing files. While we do pay for G Suite, Dropbox just seems friendlier and I don't lose any of my files. It's basically an advanced folder system for the web. Plus it's API lets me gracefully handle large files. Everything from business docs, ideas, and videos - if it's a file, it goes in Dropbox.

Slack. I know I know, some people are booing me for not using Discord. But Slack is the brand name in business communications, and Discord is known for fun and gaming. We don't pay for Slack yet, but we use it every day to communicate with each other. If you don't have a team slack, definitely try it out.

Zoom is probably the most popular program for most people these days. In a pandemic, everything goes virtual - meetings, coffees, beers, pair programming... you name it, it's likely on Zoom (or Google Meet).

Web hosting solutions in a nutshell

Kalob Taulien — Tue, 14 Dec 2021 23:02:05 +0000

There are A LOT of web hosting solutions these days.

Let's jump into a few of them, and I'll explain some of our setup at Arbington.com along the way.

Different types of hosting

There are different types of hosting. Some places give you a little space in a partitioned hard drive, some give you an entire server.

This largely depends on your budget. But you can host a website for as little as $2/month, or as much as thousands per month.

And that also, largely, depends on the complexity of your application.

Self hosted options

Personally, I like these options. These come in the form of:

AWS EC2
Digital Ocean
Linode
Heroku

There are plenty of other providers out there but those are the three most common in my circles.

You basically get an entire server for as little as $5/month. It's not actually an entire server, is a slice of a server called a "virtual machine".

AWS EC2

EC2 is, honestly, amazing! It's very powerful and very affordable. But a lot of people are trying to stay away from Amazon because EC2 hosts like, what, 30% of all websites?

But if you're an Amazon fan, EC2 is the way to go!

Digital Ocean

I'm a big fan of Digital Ocean. Simple interfaces, simple billing, powerful tools, and an overall solid company 👌

We use Kubernetes and Docker with an Nginx load balancer all hosted on Digital Ocean. That's pretty advanced, so don't feel like you need to learn everything in this paragraph to get started with Digital Ocean. They have pre-made "boxes" (virtual machines) to get you up and running with services like Django and WordPress super quickly.

Linode

I know A LOT of people that use Linode. Personally, I don't use them anymore. They are good! I just prefer Digital Ocean, that's all.

Heroku

Heroku is the odd ball here. They are incredibly powerful, very simple, with an intermediate level interface.

Less command line skills needed, less server management needed, and you can connect GitHub to it to automatically push new updates to your site. It's a dream for people that absolutely hate dealing with servers.

The common thread

You typically need to setup everything on your own, including updating Linux (minus Heroku). But with that, you get almost unlimited access to do whatever you want (minus Heroku, again).

For us at Arbington.com we use Digital Ocean - if you want a free $100 credit just to try it out, use our referral link here.

Again, this does mean you're going to learn some server ops skills. And everything is done over the command line.

So while it's only $5/month, it also means you're in full control - that's both good and bad for most people.

If you want something a little easier, I would highly recommend looking into Heroku.

It's an easier learning curve, cheap and you don't have to worry about setting up your own versions of Linux and upgrading them all the time.

The down side to Heroku is the lack of storage space - so you can't just upload all your photos and hope they'll stay there forever. You'll need something like AWS S3, Cloudflare R2, or Cloudinary to host your images.

Whereas other services like AWS or Digital Ocean give you the option to select your hard drive size (called a "disk"). If you only need 8gb, great, you only pay for 8gb. If you need 1TB, you can pay for that too (although I would suggest looking at "object stores" like AWS S3 or Digital Oceans equivalent)

Where should you start?

If you've never hosted a website before, I would suggest two things:

If your site is a static site (made with HTML/CSS/JavaScript) then look into GitHub pages.
If your site needs a backend language, try one of the easier providers that only use FTP. It's less the learn, and gets you up and running faster.

But once you have that all figured out and you feel comfortable with the basic sites, I highly recommend looking into a self hosted solution with AWS or Digital Ocean.

If you want a free $100 credit with Digital Ocean just to try things out, use our referral link.

You'll need to learn more command line, operating system and dev tools like git... but you'll also gain A LOT of experience that's super useful in "the real world", where companies are looking for these skills.

Want to learn the command line?

The command line is not to learn at first. But it's POWERFUL! And in most cases, it's faster than learning how to use a GUI (graphic user interface).

Check out these 5 command line courses on Arbington.com. Don't forget, Arbington comes with a free 14 day trial so you can try all of these courses completely risk-free.

Handling Payments with Stripe (the easy way) 💳

Kalob Taulien — Sun, 05 Dec 2021 20:22:29 +0000

At Arbington.com we use Stripe.com to handle all of our payments.

It's secure, fast, and honestly it's a developers dream.

For this, you should be familiar with Ajax/Fetch requests and being able to work with an API. We use Python on the backend, so we use Stripes's Python API.

We will show some demo code, but you'll need to fill in the blanks.

Getting started

First, you'll need a Stripe.com account. It's free to setup, and they give you two sets of API keys:

A test pair of public and private keys
A live pair of public and private keys

The test keys will make payments to your test account, and toggling between testing data and real payment data is as easy as clicking a toggle button in their dashboard.

Two things to note about Stripe:

You need a bank account attached to your Stripe account when you start accepting real data.
You never EVER send credit card numbers or CVV's (the 3 digits on the back) to your server.

How it works

On the frontend (JavaScript) you need to create a token. This token is encrypted and used by your backend so you don't ever need to send credit card information to your server.

Always. use. tokens! Never EVER send a customers raw credit card information to your server. It's unsafe and requires a thing called PCI compliance, which is a headache to obtain and maintain.

The code looks a bit like this:

<script src="https://js.stripe.com/v2/"></script>
<script>
  Stripe.setPublishableKey('your_test_publishable_key');
  const card = {
    name: nameOnCard, // Use JavaScript to get the <input> text value
    number: cardNumber,
    cvc: cardCvc,
    exp_month: cardExpMonth,
    exp_year: cardExpYear,
    address_line1: address1,
    address_line2: address2, // Optional
    address_city: city,
    address_state: state,
    address_country: country,
    currency: 'usd',
    address_zip: postalCode
  }
  Stripe.card.createToken(card, stripeResponseHandler);

  function stripeResponseHandler(status, response) {
    if (response.error) {
      // Problem!
      // Show the error
      alert(response.error.message);
    } else {
      // Token was created!

      // Get the token ID:
      const token = response.id;

      // TODO: Ajax the token to your backend/server
      // Make sure you create a POST request and not a GET request 
    }
  }

And just like that, you're creating a Stripe token for a credit card.

You don't need to save this token because a new token will be created every time you submit the payment form.

Also, on that note, don't actually wrap your html in a <form> because that could be submitted by accident and could share credit card info in a POST or GET request.

Oh, and one more thing you'll need: a JavaScript event listener. I typically tie the eventListener to a button click event, perform some light frontend validation to make sure fields aren't missing and are formatted correctly.

After you've sent the token to your backend/server, you can start to process it.

In this article I'll be using Python because it's the worlds most popular programming language, and we use it at Arbington.com.

Let's tackle that next.

Creating a payment in the backend

As mentioned, we'll be using Python. But Stripe has AMAZING documentation and AMAZING support for multiple languages, including PHP, Python, Ruby, .NET, Java, Go, and Node.js.

Let's take a look at some sample code:

import stripe
stripe.api_key = "sk_test_blah_blah_blah"

# `source` is obtained with Stripe.js this is your "token"
charge = stripe.Charge.create(
  amount=2000,  # Value in pennies, so this is 20.00
  currency="usd",
  source=request.POST.get("token"),
  description="A test token payment of $20.00 USD",
)

Pro tip: Wrap the stripe.Charge.create() method in a try/catch block.

Here's how I typically wrap a charge in Python:

try:
    charge = stripe.Charge.create(
        amount=2000,  # Value in pennies, so this is 20.00
        currency="usd",
        source=request.POST.get("token"),
        description="A test token payment of $20.00 USD",
    )
except stripe.error.CardError as e:
    # Since it's a decline, stripe.error.CardError will be caught
    ...
except stripe.error.InvalidRequestError as e:
    # Invalid parameters were supplied to Stripe's API
    ...
except stripe.error.AuthenticationError as e:
    # Authentication with Stripe's API failed
    # (maybe you changed API keys recently)
    # Log the failed transaction in the users account
    ...
except stripe.error.StripeError as e:
    # Display a very generic error to the user
    ...
except Exception as e:
    # Something else happened, completely unrelated to Stripe
    print(type(e))
    print(e)

If the charge was successful, you'll have a variable called charge (from the code above) and it'll hold A LOT of useful information you can store for later, including an id that always starts with ch_ - this can be used later for issuing refunds.

To see the entire object that Stripe returns, check out the Charge.create object in their docs

What to do next

With the data from the Charge.create object available you can save the charge id that Stripe gives you - save it in your database. This isn't considered sensitive information so you can plop it in your database without worries.

Then in your application, you can check if a user has a recent charge and possibly offer a refund - if your service offers a free trial that is.

Lastly, check your Stripe dashboard (in test mode) to see if the charge came through successfully.

If you create too much test data, you can always delete your test data in your Stripe settings. This will only effect your localhost and has no effect on your real life payments once your application is live.

Going live

Going live is SUPER easy.

If your code works in test mode, it WILL work in live mode. All you need to do it swap out your test publishable key and test private key for your live publishable key and live private key.

Remember: These keys work together. Use both testing keys together, or use both live keys together, but don't mix and match them.

And that's all there is to it.

Subscriptions are a wee bit trickier, but the concept is the same. If you'd like to learn more about how we handle subscriptions or tying a card to a specific user, let me know in the comments below!

Learning JavaScript or Python

Want to learn JavaScript or Python? Great!

Arbington.com offers over 1,500 online courses from industry experts, all for just $15/month. And it comes with a 14 day free trial.

25+ JavaScript courses

Want to learn more JavaScript? That's awesome! At Arbington we have over 25 JavaScript courses to choose from. Check them out here. These are all included in your Arbington subscription - so feel free to take them all for as low as $15/month.

40+ Python courses

Want to learn more Python? That's even more awesome! I personally LOVE Python! And at Arbington we have over 40 Python courses. These are also all included in your monthly subscription.

How we use API's at Arbington.com

Kalob Taulien — Tue, 30 Nov 2021 15:49:34 +0000

Let's talk about APIs. This is a subject I feel like every developer needs to know about.

In this article we'll be talking about the API's we use at Arbington.com, why we use them, and how they make our lives easier.

What's on the menu?

Here's a list of API's we'll talk about in this article.

Stripe for receiving payments
MailChimp and Sendy.co for sending emails
Airtable for slicing and dicing data and keeping track of tasks
Dropbox for importing videos from server to server to bypass slow user upload speeds
boto3 for AWS S3 management.
CloudFlare for video streaming

But first, what is an API?

I'm going to avoid telling you what it stands for, because it doesn't matter.

What does matter is how they work and why they exist.

So.. computers have lots of different programming languages and different ways to communicate with each other. You're likely reading this article on dev.to using a standard browser, which uses the HTTP protocol to send and receive data. (That's an oversimplification)

With all that variation from computer to computer comes the need to be able to talk to each other, make requests from other computers, and ask them to do things for us.

What you need to know.

API's have different ways to make requests. The most common for us is a RESTful API. That means we make explicit requests to a certain computer (called an endpoint) and sometimes pass data to that endpoint to guide the other computer in its task.

These often look like:

GET requests. They GET information and are typically read-only.
POST requests. They SEND information in other to create some data on another computer
DELETE requests. They DELETE information on another computer.
PUT/PATCH requests. They make updates to existing information on another computer.

I have a rad 40 minute course that goes into more depth about RESTful APIs on Arbington. Check it out here.

Payments with Stripe

Hands down the best API I've ever used. It's simple, supports lots of languages and most importantly the documentation is easy to ready and search through.

While they offer a bunch of no-code solutions, we prefer to get our hands dirty and handle payments directly. By getting in there and using the Stripe API, we can create subscriptions on the fly when a user does a thing (clicks a page or fills out a form, for instance), and modify existing prices with coupon codes.

Stripe makes life better because...
We can accept one-time payments and monthly subscriptions with just a few lines of code. And it's secure! No storing credit card information on our servers and that's great because it's one less thing we need to deal with.

Email with Mailchimp

We aren't using this A LOT but we've used it a handful of times to take lists of users that have opted in to our mailing list, and put them all into a Mailchimp list.

We do the same with Sendy.co. We also host our own campaign management system called Sendy - it uses Amazon SES and gives us 50,000 free emails per day. It comes with a very light endpoint, but not API client - so we wrote our own using Python and the requests Python package.

If you've ever gotten an email from us, chances are you went through this API process to send data from our website to our email sending website where the team can write dedicated emails and updates to you.

Mailchimp and Sendy.co makes life better because...
We can easily send emails to people expecting to hear from us.

Data analysis and collection with Airtable

My second favorite tool: Airtable. We use airtable-python-wrapper to make basic API requests to send data to Airtable for internal use (like tracking courses) and occasionally to pull data down from Airtable.

In the last post I talked about scraping 10s of thousands of data points and putting them in Airtable. Well, occasionally we need to pull that data out - so we use their API for that.

Airtable makes life better because...
It's easier than Google Sheets, has a nice API to work with huge amounts of data, and it's easier to work with the data once it's in Airtable than using Excel. (But it's pricy for big teams with big data)

Files with Dropbox

My third favorite tool, but frankly their API is overly complicated. It's ultra powerful, but not very friendly for devs that are new to the Dropbox API.

We use this to move files to and from servers.

Have you ever had to upload a video and it took AGES to finish?

We have also run into that. And it sucks. It's a lot of waiting around and slowing down your internet so you can't do very much while it's happening.

Thankfully I have gig internet - aka super fast lightning power mode internet - so I can upload insanely fast. And I put everything into Dropbox.

Using the Dropbox API my team can click through the folders (on our site, not on Dropbox.com) and import entire folders worth of content.

It creates a unique download link for each video and then a background server can download the file and upload it to wherever it needs to go. In this case, it goes to S3 or CloudFlare. More on those in just a second.

Dropbox makes life better because...
It lets us move large video files from one server to another without human interference. It makes slow uploads a non-problem for us.

More files with AWS S3

Ok, I'm not the biggest fan of S3. Frankly, it's expensive compared to other solutions. But it's also one of the oldest unlimited object stores out there and has a decent API. We use boto3 (a Python package) to upload files to S3, and to securely download them too.

Once a Dropbox file is put into S3, we get a unique URL for video. Then we use CloudFlare Stream's API to copy the video to CloudFlare Stream, where they'll transcode the video and store it for us.

S3 makes life better because...
We can store unlimited files in there and securely pull out files when we need them.

Video streaming with CloudFlare Stream

CloudFlare Stream is how we host our videos. It's powerful, relatively inexpensive and handles a lot of video encoding pain points we didn't want to handle in the early days.

And it comes with a "copy" feature - pass it a URL and it'll download the video. So we pass it an S3 or Dropbox URL and it magically appears in CloudFlare Stream.

We use plain Python and the requests library to make this happen.

Then we have a background server polling CloudFlare for updates to tell us when a video is done encoding and it's ready for playback. (Or if there was an error)

CloudFlare makes life better because...
Honestly, I'm a HUGE fan of CloudFlare. Use them for as much as you can! But CloudFlare Stream makes video encoding super easy and we don't have to worry about a custom video player.

Internal APIs

We have a lot of endpoints that can only be accessed through Ajax/Fetch requests using JavaScript. These use RESTful API methods: GET to get data, POST to create data, and DELETE to delete data.

Internal APIs make life better because...
We can use JavaScript to change/add/delete data without refreshing the page. Some pages are heavy with assets and queries, and this lets the user take an action without reloading the page.

Master APIs

I actually have a best selling course to teach you about RESTful APIs.

Learn all the things!

APIs are important to learn, but so is learning a solid programming language like Python or JavaScript (or Java, C, C#, etc).

Whichever programming language you think you should learn, take a look for those courses on Arbington.com

Up next...

Let's talk about handling payments with Stripe, and why you should 100% use Stripe instead of any other service.

Automate everything... until you can't.

Kalob Taulien — Tue, 23 Nov 2021 01:43:44 +0000

Lots of people talk about tech taking their jobs and automating humans out of a job.

And while that's sometimes true, there are a few subjects where people can't be replaced.

This is an article about how we use tech to automate big tasks at Arbington.com

But, once again, not everything can be automated (or can be automated but very poorly).

Automating big tasks

At Arbington.com we deal with a lot of scale. We have over 49,000 videos, over 4,500 hours of content, over 1,500 courses and lists of 10's of thousands of businesses, teachers and students to reach out to.

If I asked you to manually gather a list of 40,000 businesses in North America, could you compile that list with contact details in less than 3 days?

The answer to the above question is obviously no. People just can't operate that fast.

But computers can.

Using Python and less than 50 lines of code I was able to find business listing sites all over the web, scrape them, and collect data from each page.

This is all free information that's accessible to everybody. It's just A LOT of information.

And computers are good at doing one task over and over, for as long as they have reason to continue doing it. Whether that's scraping, checking if a site is down, or serving a website to thousands of views per minute.

Tools we use

Instead of hiring a small army of people to collect all the data we wanted, we wrote a small Python script that uses requests and Beautiful Soup 4 to get the page and parse the HTML. Then we used simple for loops to loop through all the data, and a handful of if statements to validate some basic data.

Then we put all that finalized data into a huge Python list and used airtable-python-wrapper,
Python and Airtable to bulk insert data into one of our tables.

The whole process took about 20 minutes to write code (and solve little bugs that came up with inconsistent data) and about 3 hours to run.

Once the code was written, we pointed it at a website and said, "Fly my beautiful butterfly, fly".

And then we waited.

If there's enough demand, maybe I'll post one of our scraping scripts to show you how we created a bot to gather and parse data, and put it into a useable table on Airtable. Leave a comment below if that interests you.

Learn Python. It's easy, and fun!

As a side note, if you're not familiar with Python you might want to consider learning it. It's the worlds #1 most popular coding language now.

And at Arbington.com we have over 35 python courses. Take them all for free with a 14 day free trial and then it's just $15/month after that. For 35 courses on Udemy that's going to be over $350. I'll let you do the math and comparison on that 😉

Human data is messy though

People, AKA: humans, are messy. We don't format phone numbers or email addresses the in a standardized way. People write:
test at gmail.com, test @ gmail . com, test [at] gmail [dot] com, and about a zillion other ways of trying to avoid spam bots from collecting their email addresses, phone numbers, business names, addresses, and so on.

First, we had to find a few directories that had relatively clean data in a consistent format.

Then we put all the data in Airtable and found the outliers with a simple filter and search. That looks like filtering for " " (a space) in an email address, for example.

Then, manually, we went and cleaned up the data so it was consistent for the entire team. From there, it became a people-focused task: emailing prospects one by one and answering questions.

We literally have hundreds of thousands of contacts now. And 95% of the data was collected through automated methods.

What would have taken a team of people MONTHS to achieve, took us a matter of days.

We still need people

Once we had all the data from our automated tasks, we needed to fill the gaps.

Not every business lists a phone number, email address or physical address. Not every portfolio website has an email address. And when they do, it's formatting incorrectly.

This is where people come in.

Instead of trying to code an automated tool that could account for every type of email address (including images) we simply said, "humans are better for this."

Humans have unparalleled pattern recognition. We can match an email address just by looking at it without reading it. Even with all the variations of email addresses out there.

So we went and manually collected the missing data.

People are simple better at tasks that involve creativity or advanced pattern recognition.

Could we have automated this? Absolutely. But it'd have taken us a long time to create a cool tool that would ultimately be thrown away.

Opportunity happens when people connect

The next big hurdle that tech couldn't solve was a simple conversation with our potential business partners.

We couldn't just email 40,000 businesses with one message. Businesses are unique, like people, and we needed to cater to each business differently.

Again, this is where people come in.

People can handle objections, consider situations, understand when to sell and when to back off, and customize each response to be friendly and helpful.

Automation can't do that.

Yes, of course it's a lot more work to hire a team of people to do this, but the end result is a much higher ROI than if we automated everything.

Automation can't replace people - that's the point I'm driving here.

When it comes to communication, creativity and empathy we simply cannot automate most of that (some exceptions apply, of course)

Should you fear losing your job as a dev, or being automated out before you land a job as a developer?

Nope. Absolutely not.

Sure, there are lots of no-code solutions out there. And when starting a startup, definitely take that approach! It'll save you a lot of pain of coding your own solutions, dealing with bugs and edge cases that users bring up.

But at the end of the day, your app/website/business has custom logic that likely can't be completely automated and will absolutely require human input.

Maybe in 2100 we'll have to open this conversation again, but for now you're safe as a developer.

For now, you can learn all your automation coding needs on Arbington.com.

💴 $15/month
🏫 14 day free trial
👩‍🏫 1,500+ courses
🔥 Certificate of Completion included in every premium courses
Check out Airtable.com - we love this service so much we don't even want to get paid for referring them. ♥️

What's next? API's!

In the next article I'll talk about the API's we use, how we use them, and why we use them. It'll be pretty dev-heavy - so if web development is your cup of tea, stick around for that one.