DEV Community: Richard Keller

Living Without Adobe Creative Suite as a Product Developer

Richard Keller — Sat, 10 Aug 2019 14:32:49 +0000

If you’re a digital product developer, you probably have Adobe software such as Photoshop and Illustrator. If you cannot live without Ubuntu as your OS, you probably don’t. Installing Adobe products on Ubuntu is a difficult process because Adobe doesn’t make Linux compatible software.

Without the Adobe software suite, I have found other ways to create/modify graphics and wireframes using open source software, freemium software, and code.

Here is a short collection of tools I have been using to replace the Adobe Suite. Some of these products are open source and some are paid subscriptions with free plans.

Whimsical.com

Flow charts, mind maps, wireframes.

Whimsical is an amazing tool for creating wireframes, flow charts, and mind maps. Whimsical has enough wireframe features to build your design without being too complicated. The best part of this app is how intuitive it is to use. Coming from using Adobe products, Whimsical is intuitive enough to get started right away without needing any tutorials.

In addition to wireframing, Whimsical allows you to create flow charts and mind maps with the same intuitive experience. This tool has been a great resource for me to design the initial prototypes of apps quickly.

Use cases:

Wireframes
Flow charts
Mind maps

Sketchpad.io

Simple graphic design.

Sketchpad.io is a decent web app for doing basic graphic design. Like Whimsical, it is very intuitive to use although there it can be buggy when using the zoom feature.

Overall, it has enough features to design quick graphics. Notable featured include: Ease of use, amazing font selection, drop in images to artboards, and easy gradient overlaying.

Use cases:

Logo/graphic design
Graphic experimentation

Gimp

Advanced graphic design.

For the seriously advanced graphic design that rivals photoshop, Gimp is the goto software for Linux (or if you are on a tight budget). Gimp is open source software that can do pretty much every you would expect from professional graphic software. The catch with Gimp is that there is a learning curve. Gimp is available for all platforms: Linux, Windows, and OSX.

Use cases:

Advanced graphic design
Logo design
Photo editing

PIL Python Image Library

Image Editing with code.

For simple image manipulation or image automation tasks, there is PIL (Python Image Library). PIL can be used to resize, filter, and crop images. Recently, I needed to make convert a white background into a transparent background. With PIL I did this with the following code:

Change a color to transparent:

from PIL import Image
def color_to_transparent():
    img = Image.open('image.png')
    img = img.convert('RGBA')
    datas = img.getdata()
    newData = []
    for item in datas:
        if item[0] < 100 and item[1] < 100 and item[2] < 100:
            newData.append((0, 0, 0, 0))
        else:
            newData.append(item)
    img.putdata(newData)
    img.save("NEW-image.png", "PNG")
    img.show()

Use cases:

Creating multiple icon versions
Color replacement
Resizing + Cropping
Automate image editing tasks

Inkscape

Vector Graphics

Finally, there is Inkscape. Inkscape is open-source software that replaces Adobe Illustrator. The interface is not as polished as Adobe Illustrator, but after studying the interface for a bit of time, it becomes easy to use, especially if you are experienced with Illustrator already.

Use cases:

Logo design
Print design
Illustrations

Conclusion

In this post, we quickly looked at some tools you can use instead of Adobe. Adobe has a great set of products, but they are not compatible with Linux. All hope is not lost though, there are many open source products that can help you accomplish your design goals.

Thanks for reading.

Anatomy of a Serverless YAML File

Richard Keller — Sun, 04 Aug 2019 16:34:17 +0000

Within the last few years, creating a serverless infrastructure has become more accessible than ever before. With AWS, Azure, and Google Cloud platforms, developers and engineers have almost limitless capabilities when deploying websites and applications. Serverless Framework, a popular framework for creating serverless architecture, has made it even easier to utilize the biggest cloud computing platforms available. They have done this by abstracting the cloud architecture configuration to a more straightforward form using YAML. In this post, we’ll look at the serverless.yml file in more detail to see how it is used to set up a serverless architecture.

Serverless.yml

The serverless.yml is the heart of a serverless application. This file describes the entire application infrastructure, all the way from the programming language to resource access.

The primary section of this YAML file is the provider. In serverless.yml configuration, you have the option to use AWS, Google Cloud, or Microsoft Azure as your serverless provider. You can also specify the programming language you want to use. In the example below, I specify that I am going to use AWS with Python 3.7.

Example serverless.yml:

service: my-first-serverless-app
provider:
  name: aws
  runtime: python3.7

The serverless.yml file above is the foundation of what could be unlimited computing power, hundreds of serverless functions, and a database infrastructure that powers your website or application. Let’s dive deeper into this configuration. I will be using AWS and python for my base example.

Environment Variables

Setting up environment variables to use in your application is really easy with the serverless.yml file. Under the provider, you add an environment section and a list of key values. Taking our example from above a step further, let’s add an environment variable that specifies a table name for a user table.

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: users_table

Now in any serverless function we use, the environment variables USER_TABLE is available for us to use. We can also use this variable in other parts of our serverless.yml file to name resources or prefix HTTP paths.

To use this environment variable in the serverless.yml later on, use the following syntax. Notice that it is similar to traversing an object in other programming languages.

${self:provider.environment.USER_TABLE}

Sourcing Other Files

Often you need to separate configuration for your application into separate files. Serverless Framework has a really nice syntax for including variables from other files. Let’s say for example that you want the user’s table to be named dev_users_table for a development branch. There are many ways to this, but to illustrate sourcing other files, I’ll show you one way.

You could have multiple config files that are used for specific environments. Let’s say you have two YAML config files in your projects root folder:

Example dev.config.yml:

table_name: 'dev_users_table'

Example prod.config.yml:

table_name: 'prod_users_table'

Your production application would use prod.config.yml. To source this prod file, you could use the following syntax:

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: ${file(./prod.config.yml):table_name}

But wait, how do you deploy this application on dev? You have to change the file to specify dev right? No, we can update this config to utilize the stage variable. The stage variable is a special variable in the Serverless Framework that can be used to specify which environment you are using. By default, the stage variable is dev.

Stage Variables

Let’s fix our serverless.yml so that we can use the dev config without updating the file manually. To do this, we use ${opt:stage, self:provider.stage}.

Example serverless.yml with stage variable:

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: ${opt:stage, self:provider.stage}_users_table

Now that we are using the stage variable. We don’t need the prod.config.yml file now. We can easily prefix all our services with the stage variable so that when we deploy our application to test a new service or feature, we don’t affect the production services.

If we did want to use separate config files we could use our stage variable to separate the configuration into separate files like this:

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: ${file(./${opt:stage, self:provider.stage}.config.yml):table_name}

Separating out your environment variables into files is a great way to organize bigger applications that use many resources such as S3 buckets, database tables, API version paths, etc.

IAM Role Statements

If you have used AWS services, you have probably had to create an IAM user or a role before. With serverless, you can give your application permissions to utilize resources via the serverless.yml file. The permission configuration is basically AWS Cloudformation written in YAML. The permissions go under the iamRoleStatements section under the provider.

Example giving your application S3 Access:

If you have an S3 bucket named awesome-bucket-name. The following example will give your application full access (Read, Write, List, etc) to the specified bucket.

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: users_table 
 iamRoleStatements:
    -  Effect: "Allow"
       Action:
         - "s3:*"
       Resource:
          - arn:aws:s3:::awesome-bucket-name

Example giving your application SES:

If you have an application that needs to send emails. You can give it access to Amazon SES like this: (Note: you need to setup SES with a verified domain separately.) The following example gives your application access to use all SES resources to send emails.

provider:
  name: aws
  runtime: python3.7
  environment:
    USER_TABLE: users_table 
 iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "ses:SendEmail"
      Resource: "*"

Functions

I’ve been writing about this mysterious “application”, but I haven’t explained what that is. In your serverless.yml file, you configure functions to be created on your serverless provider’s platform. For AWS, you can tell serverless to create Lambda functions that can be routed to HTTP endpoints. Let’s take a quick look at how to define an HTTP endpoint via the serverless.yml file.

Without functions, you do not actually have an application. Remember the programming language we specified in the provider? That tells our provider which programming language to create our functions with. (Note: Programming languages are determined by the provider. You cannot pick any language.)

As an example, let’s create an HTTP endpoint that returns the md5 hash of an email address, along with the Gravatar URL for that email.

To define a function we need two things:

A file with the function called getHash we’ll call the file handler.py
A serverless.yml function definition

Example of the serverless.yml definition:

functions:
  myCustomFunctionName:
    handler: handler.getHash

On the first line, we have functions. This is a section of the YAML file that lives under the provider section. In the functions section, we have myCustomFunctionName. This name will be the Lambda function name in AWS Lambda, not the one used in the code.

Next, we define the handler. The handler is the path to the function using dot notation. This dot notation is similar to how imports work in Python. handler is the file name and getHash is the function name in that file.

Here is our handler.py:

import hashlib
import json
CORS = {
    'Access-Control-Allow-Origin': '*',
    'Access-Control-Allow-Credentials': True,
}
def getHash(event, context):
    string = event['pathParameters']['string']
    h = hashlib.md5(string.encode('utf8'))
    h = h.hexdigest()
    return {
        "statusCode": 200,
        "headers": CORS,
        "body": json.dumps({
            "hash": h,
            "avatar": "http://gravatar.com/avatar/{}".format(h)
        })
    }

With the file defined and the function named getHash inside, our serverless.yml is ready to create our function on AWS Lambda. Before we deploy this, we need to enable access to this function via HTTP. To make this an HTTP endpoint, let’s add an events section.

Example functions with events:

functions:
  myCustomFunctionName:
    handler: handler.getHash
    events:
      - http:
          path: md5/{email}
          method: get
          cors: true

The event section ties our function to the API Gateway provided by AWS. In this section, we specify the path and the method of the HTTP endpoint. With this event configuration, you will be able to access your Lambda function via the path /md5/some@email.com. The response would be a JSON object with the md5 hash and a URL to the gravatar image.

Example output with my email:

{
  "hash": "97fbef641c6e4669b6a1ad4ffb3342f5",
  "avatar": "http://gravatar.com/avatar/97fbef641c6e4669b6a1ad4ffb3342f5"
}

You can go even further with this by requiring authentication and rate-limiting this endpoint. More from the docs.

Now that we have our events, we can send get requests to this function from a frontend application and it will run our serverless function.

Conclusion

This post scratches the surface of the serverless.yml file. I hope you understand a little more about serverless and dive deeper into this framework and technology. Serverless Framework, along with the cloud platforms available, empower people to create scalable products at a low cost. Building the infrastructure for an application is much easier than ever before and can be created by using a simple configuration file like serverless.yml.

Thanks for reading!

Why you should not build your own authentication system?

Richard Keller — Sat, 27 Jul 2019 23:00:20 +0000

Now I know why people always say: “don’t roll your own authentication system.” Unless you want to create an authentication system for fun, I would advise against it.

Complexity

First of all, it’s a full-time job to create a robust authentication system. For example, you need many API routes, knowledge of encryption, a server to host the API, and a database server. That is just the minimum. Authentication systems are much more complicated than they seem.

Security & Multi-level dependencies

You also need to keep up with security patches for your authentication systems application dependencies. Also, you need to keep up with the dependencies of the software you are using to build your application.

The Wheel argument

Another reason to avoid creating an authentication system is that there are already great ones available. Just take your pick:

Auth0
Firebase
Google Login
Facebook Login
Twitter Login
Github Login

All of these applications can be used as an authentication system for your application for minimal cost or free.

Give it a try

For me, I’ve always wanted to write an authentication system from scratch, and so I have written a few. It is a good idea to create an authentication system as a programming exercise, but you should use an existing authentication system for a production application.

The two reason to create your own authentication system:

Programming exercise
You want to build an authentication SaaS product.

To conclude, you should “roll-your-own auth.” It’s usually very complex, there are many security dependencies to worry about, and there are already great solutions available. So build your application and integrate it with an existing authentication system. You’ll thank me later!

Crime Data Exploration with Python3 and Pandas: Part 2

Richard Keller — Tue, 23 Jul 2019 02:16:35 +0000

Objective

The purpose of this post series is to explore datasets from the UCR (Uniforn Crime Reporting Program). In this post, we'll use Pandas to dig deeper into crime statistics for homicide. After preparing the dataset for comparison, we'll rank each state based on number of crimes per capita.

Follow along or checkout the code on Github.

About the data

The dataset we are using is the estimated_crimes.csv downloaded from the Cime Data Explorer website. This file contains the estimated crimes for 7 types of crimes, from the years 1995 to 2017, for the United States. For more information about this dataset and how it is compiled visit the Cime Data Explorer.

Preparing the data

For data manipulation we will use Pandas. The first thing we need to do is download the dataset and load it into a Pandas dataframe.

If you need the dataset you can download it here: https://github.com/rbk/Crime-Data-Analysis

In the following code, we use pandas to open the estimated_crimes.csv into a dataframe.

We specify the columns we want to work with by specifying the usecols attribute. This attribute tells the read_csv function to only load in the column with the headers that are in the cols array:

import pandas

cols = [
    'year',
    'state_name',
    'population',
    'homicide',
]

raw_data = pandas.read_csv('../data/estimated_crimes.csv', usecols=cols)

print(raw_data.head())

Output:

   year state_name  population  homicide
0  1995        NaN   262803276     21606
1  1996        NaN   265228572     19645
2  1997        NaN   267783607     18211
3  1998        NaN   270248003     16974
4  1999        NaN   272690813     15522

If you are curious on how to find out what the headers of your CSV are, use the info function:

data = pandas.read_csv('../data/estimated_crimes.csv')
print(data.info())

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1196 entries, 0 to 1195
Data columns (total 16 columns):
year                   1196 non-null int64
state_id               1173 non-null float64
state_abbr             1173 non-null object
state_name             1173 non-null object
population             1196 non-null int64
violent_crime          1196 non-null int64
homicide               1196 non-null int64
rape_legacy            1196 non-null int64
rape_revised           260 non-null float64
robbery                1196 non-null int64
aggravated_assault     1196 non-null int64
property_crime         1196 non-null int64
burglary               1196 non-null int64
larceny                1196 non-null int64
motor_vehicle_theft    1196 non-null int64
caveats                71 non-null object
dtypes: float64(2), int64(11), object(3)

Top 10 States Ranked By Total Homicides

Now that we have our data loaded into a dataframe, we can start processing the data. The first thing we will do is rank the states by the number of homicides.

This is a bad idea. Let's do it anyway to see why.

This is a very simple task. First, we query the dataframe by the year:

# Get 2017 state data
# We use notnull because the state name is not null for states
data = raw_data.query('year == 2017 and state_name.notnull()')

Next we sort the data using sort_values.
Using the by attribute, we provide an array of columns to sort by. In this case we are sorting by the count of homicides.

# Ranked By Total Homicides
ranked_by_total = data.sort_values(by=['homicide'], ascending=False)
ranked_by_total = ranked_by_total.reset_index()
del ranked_by_total['index']
print(ranked_by_total.head(10))

Output:

   year      state_name  population  homicide
0  2017      California    39536653      1830
1  2017           Texas    28304596      1412
2  2017         Florida    20984400      1057
3  2017        Illinois    12802023       997
4  2017    Pennsylvania    12805537       739
5  2017            Ohio    11658609       710
6  2017         Georgia    10429379       703
7  2017        Missouri     6113532       600
8  2017  North Carolina    10273419       591
9  2017       Louisiana     4684333       582

The homicide column above is sorted by count.

From this simple sort we can say that California has the most homicides per year. Statistically this is true, but they also have the largest population of any state.

Top 10 States By Population

   year      state_name  population  homicide
0  2017      California    39536653      1830
0  2017           Texas    28304596      1412
0  2017         Florida    20984400      1057
0  2017        New York    19849399       548
0  2017    Pennsylvania    12805537       739
0  2017        Illinois    12802023       997
0  2017            Ohio    11658609       710
0  2017         Georgia    10429379       703
0  2017  North Carolina    10273419       591
0  2017        Michigan     9962311       569

Ranking the states in this way doesn't make sense because the number of homicides is not proportional to the number of people in the states.

From looking at the homicides ranked vs the populations, you can see that there is a strong correlation between the number of homicides and the population.

Next, we will rank the states by homicide rate.

Top 10 States Ranked By Total Homicides Relative to the Population Size

Now we will rank the states in homcide by the population size. This will give us a clearer picture of the homcide rate, e.g. the number of homicides per 100,000 people.

First, we'll use Pandas apply function to create a new row.
The apply function takes the name of a function and "applies" the result of the function to each row. Our function is called per_capital.

The per_capita function takes each row of data and performs a calculation to normalize data.

The result is that we have a column for each row with the number of homicides per 100,000 people.

# Ranked By Total Homicides Relative to the Population Size
# Per 100,000 people
def per_capita(row):
    """Calculate the homcide rate per capita."""
    total_homicides = row['homicide']
    population = row['population']
    count = (total_homicides / population) * 100000
    return count

data['per_captia'] = data.apply(per_capita, axis=1)

# Ranked By Total Homicides
ranked_by_population = data.sort_values(by=['per_captia'], ascending=False)
ranked_by_population = ranked_by_population.reset_index()
del ranked_by_population['index']
print(ranked_by_population.head(50))

Finally, we print all the rows.

Output:

    year            state_name  population  homicide  per_captia
0   2017  District of Columbia      693972       116   16.715372
1   2017             Louisiana     4684333       582   12.424394
2   2017              Missouri     6113532       600    9.814294
3   2017                Nevada     2998039       274    9.139307
4   2017              Maryland     6052177       546    9.021547
5   2017              Arkansas     3004279       258    8.587751
6   2017                Alaska      739795        62    8.380700
7   2017               Alabama     4874747       404    8.287610
8   2017           Mississippi     2984100       245    8.210181
9   2017             Tennessee     6715984       527    7.846951
10  2017              Illinois    12802023       997    7.787832
11  2017        South Carolina     5024369       390    7.762169
12  2017            New Mexico     2088070       148    7.087885
13  2017               Georgia    10429379       703    6.740574
14  2017              Oklahoma     3930864       242    6.156407
15  2017                  Ohio    11658609       710    6.089920
16  2017               Indiana     6666818       397    5.954865
17  2017               Arizona     7016270       416    5.929076
18  2017              Kentucky     4454189       263    5.904554
19  2017          Pennsylvania    12805537       739    5.770941
20  2017        North Carolina    10273419       591    5.752710
21  2017              Michigan     9962311       569    5.711526
22  2017              Delaware      961939        54    5.613662
23  2017                Kansas     2913123       160    5.492387
24  2017              Virginia     8470020       453    5.348275
25  2017               Florida    20984400      1057    5.037075
26  2017                 Texas    28304596      1412    4.988589
27  2017         West Virginia     1815857        85    4.680985
28  2017            California    39536653      1830    4.628616
29  2017              Colorado     5607154       221    3.941393
30  2017               Montana     1050493        41    3.902929
31  2017            New Jersey     9005644       324    3.597744
32  2017                  Iowa     3145711       104    3.306089
33  2017             Wisconsin     5795483       186    3.209396
34  2017            Washington     7405743       230    3.105698
35  2017          South Dakota      869666        25    2.874667
36  2017           Connecticut     3588184       102    2.842664
37  2017              New York    19849399       548    2.760789
38  2017                Hawaii     1427538        39    2.731976
39  2017               Wyoming      579315        15    2.589265
40  2017         Massachusetts     6859819       173    2.521932
41  2017                Oregon     4142776       104    2.510394
42  2017                  Utah     3101833        73    2.353447
43  2017               Vermont      623657        14    2.244824
44  2017              Nebraska     1920076        43    2.239495
45  2017             Minnesota     5576606       113    2.026322
46  2017          Rhode Island     1059639        20    1.887435
47  2017                 Idaho     1716943        32    1.863778
48  2017                 Maine     1335907        23    1.721677
49  2017          North Dakota      755393        10    1.323814
50  2017         New Hampshire     1342795        14    1.042601

What can we infer from this analysis

The new column per_capita, gives us a more accurate description of the homicide rate per 100,000 people for each state. For example, we could say that for every 100,000 people in Louisiana, there are 2.4 homicides reported.

Here are a few more conclusions we can draw from this analysis:

The District of Columbia has the highest homicide rate, ranking #1 for the most homicides per capita (per 100,000 people).
Although California has the largest population, the homicide rate ranks #29 in the country.
New Hampshire has the lowest homicide rate.

Conclusion

The lesson from this analysis is that sorting by count doesn't tell the full story about the data. California would be ranked #1 in homicides from the first analysis, but in reality, California has a lower homicide rate than 28 other states. This is a simple example of how data can be decieving.

To recap, we used Pandas read_csv to explore the Estimated Crime 2017 dataset. We ranked states by total homicides. Then we looked at the homicide rate, which gave of very different views of how the states rank in homicdes.

Thanks for reading!

FBI Disclaimer

"The data found on the Crime Data Explorer represents reported crime, and is not an exhaustive report of all crime that occurs. It’s important to consider the various factors that lead to crime activity and crime reporting in a community before interpreting the data. Without these considerations the available data can be deceiving. Factors to consider include population size and density, economic conditions, employment rates, prosecutorial, judicial, and correctional policies, administrative and investigative emphases of law enforcement, citizens’ attitudes toward crime and policing, and the effective strength of the police force."

--- Crime Data Explorer, Retrieved 19:45, July 22, 2019

For more information on the use of the UCR database

Crime Data Exploration with Python3 and Pandas: Part 1

Richard Keller — Sat, 13 Jul 2019 21:48:40 +0000

Overview

Data is everywhere. Crime is too. The FBI runs a program called Uniform Crime Reporting (UCR) Program that collects crime data from ~18,000 agencies all over the United States. This database is a gold mine of interesting statistics about crime in the United States. Follow along and learn about how you can use Python to analyze this data. My hope is that you learn from this article and then go do some data exploration on your own.

Full Source code: https://github.com/rbk/Crime-Data-Analysis

Data Tools We'll Use

Data scientists are known to use Python for machine learning and data cleaning. What you may not know is that there are some fantastic libraries in Python for performing operations on JSON, CSV, and other data types.

You are going to fall in love with Pandas very soon. No not the cute cuddly pandas you see at the zoo, Pandas the Python package. Pandas is a library that makes dealing with massive arrays of data a breeze as you will see in the next section. Pandas is built on top of Numpy. I recommend checking them both out:

Download the Dataset

Grab the dataset to begin. You can use which ever method is easiest to you to download the dataset. Here is the URI of the dataset we will be working with:

http://s3-us-gov-west-1.amazonaws.com/cg-d4b776d0-d898-4153-90c8-8336f86bdfec/estimated_crimes.csv

This dataset contains the esitmated counts of criminal incidents reported from 1995 to 2017 in all the United States.

Before diving in to the code there is one more thing...disclaimer time.

Disclaimer About Drawing Conclusions from this Crime dataset

The crime dataset is a compilation of incident reports over time. The term "Crime Rate" is the number of crimes per person in a given population. It is important to note that this doesn't take into account the hundreds of risk factors associated with the safety of a city or state. For example, New Mexico may have a higher crime rate than average, but statistically that doesn't New Mexico is a dangerous place to live. The data doesn't take into account the population density, economic conditions, employment rates, and the effective strength of the police force¹.

Furthermore, you cannot just assume that a certain state is more safe because the crime rate is below average. The reason this claim is invalid is that crime in certain areas could just not be reported because of the crime itself.

Alright, enough of the disclaimer...

Time to Code!

Fire up your IDE or editor or Jupyter Notebook ladies and gents. We are going in line by line. You should have a folder with a CSV file called estimated_crimes.csv. In this folder you need a Python file. I usually use main.py. This code will be in Python3. You'll also need to pip install pandas:

pip install pandas

Objective

In the following steps we will transform the CSV file from the FBI into a new CSV with new columns that help us visually understand the data. Specifically, we are going to determine the average crime rate on the national level and then compare each US state's crime rate to show the percentage difference relative to the national average.

Step 1: Load the CSV into a Pandas Dataframe

The first thing we need to do is get control of our data. In its raw format it is a little awkward to work with. Pandas is the tool of choice for transforming data into different forms that are easier to work with.

The first trick you must know about Pandas when working with CSV files is the amazing read_csv function. The read_csv function will load any CSV file into a Pandas Dataframe. From this dataframe we can use all kind of pandas functions to manipulate and query the data.

import pandas

# Read the CSV into a Dataframe
data = pandas.read_csv('estimated_crimes.csv')

Once the data is read into a variable we can use the head function to print the first 5 rows of the dataframe:

# Print the first 5 lines of the dataframe
print(data.head())

Output

Now before moving on, lets remove the columns of the data we don't need right now. To do this, read the CSV with the usecols option. usecols tells the read function to only use a specified list of columns:

import pandas

# Which columns to use
columns = ['year', 'state_name', 'population', 'violent_crime', 'property_crime']

# Get the data into a dataframe from csv
data = pandas.read_csv('estimated_crimes.csv', usecols=columns)

Now when you print the data it is easier to visualize.

# Print the first 5 lines of the dataframe
print(data.head())

Output

Step 2: Filter Data

Now that you have the data in a dataframe you have unleashed the power of Pandas! The second trick of Pandas is the query function. With the query function we can filter our CSV file to only contain rows for a specific year.

Using our dataframe from above:

data = data.query('year == 2017')

If that doesn't make you excited to use pandas I don't know what will. After running this query function our data should only contain rows in which the year column is 2017.

Another filter we need to add is to remove row in which the state column is null. In this particular CSV the top rows are the compiled national stats, so the state column is not filled out. We can filter these rows by using the pandas notnull function:

data = data[data['state_name'].notnull()]

Any row in which the state column is null or NaN, will be removed from the dataframe.

Step 3: Add new data columns with `apply`

Now that we have the data cleaned up. We'll add three columns. Using the pandas apply function we can create new columns in the data in two steps.

The first step is to define your function. This function will take each row from the dataframe. For example, the following function will calculate the crimes per 1000 people in each row.

def crime_per_capita(row, number_of_people):
    total_crimes = row['violent_crime'] + row['property_crime']
    population = row['population']
    count = (total_crimes/population)*number_of_people
    return count

Our dataframe can now use this function to create a new column by using apply:

data['crime_per_1000'] = data.apply(crime_per_capita, args=(1000,), axis=1)
print(data.head())

Running print head will show a new column with the number of crimes for every 1000 people.

Next we will compute the national crime rate, then use the national crime rate to compare each individual states crime rate. Once we compute the percentage difference of the national average crime rate to each US State, we will be able to quickly see the differences in crime rates across the US.


# 1 Get National Average
total_population = data['population'].sum()
total_crimes =+ data['violent_crime'].sum() + data['property_crime'].sum()
national_average_per_cap = (total_crimes/total_population)*1000

# 2.0 Add column of diff to national average
def compute_diff(row):
    diff = row['crime_per_1000'] - national_average_per_cap
    return diff

data['diff_of_national_average'] = data.apply(compute_diff, axis=1)

# 2.1 Add column of percent diff compared to national average
def compute_percent(row):
    percent = (row['crime_per_1000']/national_average_per_cap)*100
    if percent > 100:
        return "+" + str(int(percent-100)) + "%"
    return "-" + str(100 - int(percent)) + "%"

data['percent_diff_national'] = data.apply(compute_percent, axis=1)

Drawing Conclusions

From the data above we can see that:

Ohio and Maryland have the lowest crime rates 2% below the average. This means that there we fewer crimes reported for every 1000 people than the national average.
Maryland has the fewest crimes per capita in 2017.
The crime rate in the District of Columbia is nearly twice the national average (+91% above the national average).

The possibilities of data exploration are endless with this dataset and many others.
Just keep in mind that you should take heed to the disclaimer before making assumptions about safety on just one dimensinal data like this. There are hundreds of factors that determine the actual safety of a city or neighborhood.

To conclude, we looked at a dataset from the FBI UCR website called "estimated_crimes.csv". We used the Pandas python library to filter and transform the data so that it could be read easier. We then came up with a few conclusions about the data.

Stay tuned for Part 2 where we dive deeper into the Crime datasets.

Resources and References

[1] https://crime-data-explorer.fr.cloud.gov/explorer/state/california/crime/1995/2017
https://www.fbi.gov/services/cjis/ucr
https://crime-data-explorer.fr.cloud.gov/downloads-and-docs
https://definitions.uslegal.com/c/crime-rate/ -> "Population-based rates fail to take into account variations in risk."

Building a PDF Generator on AWS Lambda with Python3 and wkhtmltopdf

Richard Keller — Mon, 24 Jun 2019 12:09:19 +0000

Update 07/18/2019: Source Code: https://github.com/rbk/Lambda-PDF-Generator

UPDATE 06/28/2019: For anyone attempting to follow this please read a follow up post about a font bug on the generated PDFs: https://blog.richardkeller.net/wkhtmltopdf-bad-kerning-on-aws-lambda/

Introduction

Creating scalable APIs in 2019 is easier than ever before with serverless auto-scaling compute power being widely accessible. In this article I'll show you how I created a PDF generator API that can handle with 1000 concurrent requests.

The purpose of this artice is to show you how I accomplished creating the API so that you will see how easy it is to create your own serverless APIs. When this project is complete, you will have an API endpoint in which you can POST a JSON object to, and recieve a link to a PDF on Amazon S3. The JSON object looks like this:

{
  "filename": "sample.pdf",
  "html": "<html><head></head><body><h1>It works! This is the default PDF.</h1></body></html>"
}

Setup

What you'll need:

Python3
Aws CLI installed and configured
Serverless installed

Serverless

The first thing you need to do is initialize a new Serverless project. Serverless is a tool that greatly simplifies using AWS, Gcloud, and Azure services so you can create APIs without worrying about managing a server. If you need more information about Serverless visite their website: Serverless.com. In your terminal run the following command to initialize your project.

Note: sls is short for serverless and can be used interchangably.

sls create --template aws-python3

This command will bootstrap a Python3 Lambda setup for you to work from.

WKHTMLTOPDF Binary

Next we need to get the binary for turning HTML into PDFs. For this I used WKHTMLTOPDF. Important: You have to use version 0.12.4 becaused later versions of wkhtmltopdf require dynamic dependencies of the host system that cannot be installed on Lambda.

Download version 0.12.4 here:

https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz

Once you have extract this tar file, copy the binary wkhtmltopdf to the binary folder in your project.

./binary/wkhtmltopdf

More information about wkhtmltopdf can be found on their website:
WKHTMLTOPDF

Python3 Dependencies

Now that we have the WKHTMLtoPDF binary, we need the Python library, pdfkit to use it. Since we are using Serverless and AWS Lambda we cannot just run pip install pdfkit. We need a Serverless plugin to install our dependencies on Lambda.

In our project folder install the python plugin requirements module for Serverless.

sls plugin install -n serverless-python-requirements

Now, in your serverless.yml file, you need to add a custom section in the yaml:

custom:
  pythonRequirements:
    dockerizePip: true

Once the serverless plugin requirements is installed, you can add a requirements.txt file to your project and it will be automatically installed on lambda when you deploy.

Your requirements.txt for this project only needs to have pdfkit.

Requirements.txt

pdfkit

For any issues with this module checkout the repository issue:
Serverless Python Requirements

Ready, Set, Code

We are ready to code now. We only need to work with two files for this project: serverless.yml ad handler.py. Typically you want your serverless projects to be lightweight and independent of other parts of your codebase.

Serverless.yml

The serverless.yml file is our main configuration file. A serverless.yml file is broken down into a few important sections. If you are new to YAML, checkout the documentation for YAML.

For this serverless.yml configuration you'll need to create a file called config.yml. This will store the S3 bucket name. The serverless.yml will reference the config.yml to setup the correct bucket for your project.

Contents of config.yml

BucketName: 'your-s3-bucket-name'

Here is a high level overview of a serverless.yml file:

service: pdf-services # name or reference to our project
provider: # It is possible to use Azure, GCloud, or AWS
functions: # Array of functions to deploy as Lambdas
resources: # S3 buckets, DynamoDB tables, and other possible resources to create
plugins: # Plugins for Serverless
custom: # Custom variables used by you or plugins during setup and deployment

Our serverless configuration will do a few things for us when we deploy:

Create an S3 bucket called pdf-service-bucket to store our PDFs
Create a function that will create the PDFs
Give our function access to the S3 bucket
Setup an API endpoint for our Lambda function at:

POST https://xxxx.execute-api.xxxx.amazonaws.com/dev/new-pdf

Here is the full serverless.yml configuration. I've added a couple important comments in the code.

service: pdf-service
provider:
  name: aws
  runtime: python3.7
  # Set environment variable for the S3 bucket
  environment:
    S3_BUCKET_NAME: ${file(./config.yml):BucketName}
  # Gives our functions full read and write access to the S3 Bucket
  iamRoleStatements:
    -  Effect: "Allow"
       Action:
         - "s3:*"
       Resource:
          - arn:aws:s3:::${file(./config.yml):BucketName}
          - arn:aws:s3:::${file(./config.yml):BucketName}/*
functions:
  generate_pdf:
    handler: handler.generate_pdf
    events:
      - http:
          path: new-pdf
          method: post
          cors: true
resources:
 # Creates an S3 bucket in our AWS account
 Resources:
   NewResource:
     Type: AWS::S3::Bucket
     Properties:
       BucketName: ${file(./config.yml):BucketName}
custom:
  pythonRequirements:
    dockerizePip: true
plugins:
  - serverless-python-requirements

Handler.py

Handler.py is the only Python file we need. It contains one function to generate the PDF and save it to Lambda. You can creat more functions in this file to split the code into reusable parts, but for this example one function was enough. Lambda functions take two arguments by default: Event and Context.

Context contains environment variables and system information.

Event contains request data that is sent to the lambda function.

In this project we will send our function generate_pdf a filename and HTML, and it will return the URI for a PDF it creates.

import json
import pdfkit
import boto3
import os
client = boto3.client('s3')

# Get the bucket name environment variables to use in our code
S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME')

def generate_pdf(event, context):

    # Defaults
    key = 'deafult-filename.pdf'
    html = "<html><head></head><body><h1>It works! This is the default PDF.</h1></body></html>"

    # Decode json and set values for our pdf    
    if 'body' in event:
        data = json.loads(event['body'])
        key = data['filename']
        html = data['html'] 

    # Set file path to save pdf on lambda first (temporary storage)
    filepath = '/tmp/{key}'.format(key=key)

    # Create PDF
    config = pdfkit.configuration(wkhtmltopdf="binary/wkhtmltopdf")
    pdfkit.from_string(html, filepath, configuration=config, options={})


    # Upload to S3 Bucket
    r = client.put_object(
        ACL='public-read',
        Body=open(filepath, 'rb'),
        ContentType='application/pdf',
        Bucket=S3_BUCKET_NAME,
        Key=key
    )

    # Format the PDF URI
    object_url = "https://{0}.s3.amazonaws.com/{1}".format(S3_BUCKET_NAME, key)

    # Response with result
    response = {
        "headers": {
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Credentials": True,
        },
        "statusCode": 200,
        "body": object_url
    }

    return response

Deploy

Now that you have all the code ready it is time to deploy. In your project folder run the following command from your terminal:

sls deploy

After you run deploy, Serverless will create everything for you in AWS. You will get HTTP POST endpoint that you will use to generate PDFs. The endpoint will look something like this:

https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/new-pdf

You can use curl to test your function. The following curl command posts a JSON object to the lambda endpoint. The JSON object contains a filename and some html to turn into a PDF.

curl -d '{"filename":"my-sample-filename.pdf", "html":"<html><head></head><body><h1>Custom HTML -> Posted From CURL as {JSON}</h1></body></html>"}' -H "Content-Type: application/json" -X POST REPLACE-WITH-YOUR-ENDPOINT

Note: Replace "REPLACE-WITH-YOUR-ENDPOINT" with the endpoint you receive from Serverless.

After running this command you should receive the URI to your generated PDF.

Conclusion

Creating a scalable PDF generator is easy with of AWS and Serverless. In addition to having a scalable API, you don't have to worry about server maintenance. I encourage you to create more projects with AWS Lambda to get comfortable with the platform and configuration.

Thanks for reading!

Next Steps

Setup a custom domain with Serverless Domain Manager
Setup local testing with Serverless Offline

Resources

Serverless Environment Variables - https://serverless.com/framework/docs/providers/aws/guide/variables/
Serverless CORS Guide - https://serverless.com/blog/cors-api-gateway-survival-guide/
Fonts AWS Lambda - https://forums.aws.amazon.com/thread.jspa?messageID=776307
Boto3 API - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.upload_file
S3 Argument Reference - https://gist.github.com/rbk/926bfd3d886b2c25c53818eeb6e77d6a

Originally published at https://blog.richardkeller.net.

What I Learned From Tulsa's Web Developer Conference 200OK

Richard Keller — Mon, 10 Jun 2019 12:02:18 +0000

This article is a collection of thoughts from my experience attending Tulsa's Web Developer Conference 200Ok in 2019. I would Like to point out that all of the talks at the conference were interesting and helpful in there own way. This article will just cover the parts I found most relevant.

There were three talks at the 200OK conference that really stood out to me. The three talks were about having the growth mindset, writing good APIs, and efficiency through kanban. I believe these three things are fundamental to being a good web developer. Let’s explore these areas briefly, one by one.

Have a Growth Mindset

The first talk I want to mention was a talk from Christine Seeman, a Software Engineer. Her talk was titled “Hack your brain – Ways to improve your thought processes.” The talk was really awesome and included a meditation session which made it even better.

It is important for web developers to have a growth mindset. Having a growth mindset means that you look at problems as learning opportunities. There is always something new to learn in web development because the technology changes fast. The pace of technology can be overwhelming. Here are a couple tips I learned from this talk about having a growth mindset.

1 Accept that you are not perfect

This is a big one. You need to be humble since you cannot possibly know everything. It is important to accept that you are not perfect. Everyone wants to be the best at their job, which is understandable, but you need to also realize that you have limited time to learn and so you cannot be an expert at everything.

2 Focus on the learning process, not the end result

The end result looks great, but the process of learning is more important to focus on. You need to figure out and understand how you learn best, to help yourself learn better and faster. There are tons of resources online in which you can find out how to learn better. Here is a free online course that explains techniques like spaced repetition, recall, and visualizaton to help you learn and retain information better: Learning How To Learn, https://www.coursera.org/learn/learning-how-to-learn.

APIs Should Be Documented and Consistent

The second talk I want to mention was from Michele Titolo, a Lead Engineer. Her talked was titled **”APIs: The good the bad the ugly.”**about the good and bad parts of REST APIs. I just want to mention a few things about what makes a good API. A good API has many characteristics, but the three characteristics that stand out to me the most are consistency, semantic, and documented.

1 Documented

Web developers usually loath writing documentation which is odd because they love consuming great documentation. Documentation is actually just one part of being a good web developer. Helping others understand how to use your code is essential for that code base to survive and thrive. An API that isn’t documented will probably never be used because it is not easy to understand and API from just looking at code alone.

Good documentation of your API makes it better automatically by being more accessible. Here is a template I created for documenting a REST API as an example:

https://gist.github.com/rbk/c6d2e80c9f4830857cafa3d24e97e227

2 Consistent

Consistency of an API super important. Error codes, request payloads, and response types can be anything you want, just make sure it makes sense. This is important for code in general. You don’t want to mix JSON responses with XML responses. One way I have made error codes standard is by creating a standard object that follows the API guidelines from Microsoft (Ironically they don’t always use this guide):

payload = {
    'error': {
        'code': 'InvalidParameter',
        'message': 'You sent the wrong parameter.'
    }
}
return Response(payload, status=406)

Kanban is All About Flow Efficiency

The last talk I want to mention was from Paul Gower. His talk was titled “Overcoming Delay – How The Best Devs Increase Productivity”.

I have heard the word Kanban, but I thought it meant moving todos from column to column. This talk opened my eye to the world of Kanban. Kanban is essentially about efficiency.

1 Flow Efficiency

Kanban is not magic, it is a process to increase efficiency. The basic concept can be examined in “Little’s Law.” The output stream is greater or equal to the input stream of work. This means having a stream of production move through an organization is imperative to satisfy customers on the other end of the production line to grow the business.

2 WIP (Work in progress)

Kanban increases efficiency with the use of controlling WIP (Work in Progress). This control is intended to keep team from being overwhelmed with too much work like emails, questions, and tasks, etc that could decrease individual productivity.

Conclusion

Tulsa’s 200OK Web Developer Conference is always a blast to attend. The conference always supplies me with relevant information that I can use in my day to day career as a Web Developer. I hope you found some of these tips and bits useful. Thanks for reading.

BONUS: Notes from the conference.

Being a developer is not about money, it’s about solving problems.
Developers need to be disciplined to finish projects.
Algorithms and programming are two ways to express the same solution to a problem.

DEV Community: Richard Keller

Living Without Adobe Creative Suite as a Product Developer

Whimsical.com

Sketchpad.io

Gimp

PIL Python Image Library

Inkscape

Conclusion

Anatomy of a Serverless YAML File

Serverless.yml

Environment Variables

Sourcing Other Files

Stage Variables

IAM Role Statements

Functions

Conclusion

Why you should not build your own authentication system?

Complexity

Security & Multi-level dependencies

The Wheel argument

Give it a try

Crime Data Exploration with Python3 and Pandas: Part 2

Objective

About the data

Preparing the data

Output:

Output:

Top 10 States Ranked By Total Homicides

Output:

Top 10 States Ranked By Total Homicides Relative to the Population Size

Output:

What can we infer from this analysis

Conclusion

FBI Disclaimer

Crime Data Exploration with Python3 and Pandas: Part 1

Overview

Data Tools We'll Use

Download the Dataset

Disclaimer About Drawing Conclusions from this Crime dataset

Time to Code!

Objective

Step 1: Load the CSV into a Pandas Dataframe

Step 2: Filter Data

Step 3: Add new data columns with apply

Drawing Conclusions

Resources and References

Building a PDF Generator on AWS Lambda with Python3 and wkhtmltopdf

Introduction

Setup

Serverless

WKHTMLTOPDF Binary

Python3 Dependencies

Ready, Set, Code

Serverless.yml

Handler.py

Deploy

Conclusion

Next Steps

Resources

What I Learned From Tulsa's Web Developer Conference 200OK

Have a Growth Mindset

1 Accept that you are not perfect

2 Focus on the learning process, not the end result

APIs Should Be Documented and Consistent

1 Documented

2 Consistent

Kanban is All About Flow Efficiency

1 Flow Efficiency

2 WIP (Work in progress)

Conclusion

BONUS: Notes from the conference.

Step 3: Add new data columns with `apply`