DEV Community: Wilson

Five Things I Learned From Making NYC Arbor Logger

Wilson — Mon, 04 May 2020 19:03:43 +0000

Five Things I Learned From Making NYC Arbor Logger

Overview

Estimated Reading Time: 4.5 minutes

I recently finished a simple full stack project and learned a good amount from it! I decided to share a couple of things that I think might help you on your developer journey.

For some background on this project, I wanted to build a full stack project with a Flask backend, all while using open source data to make data visualizations on a React frontend. The scope of this project wasn't super large and since I didn't have to do any data analysis, processing the data is straight forwards.

That said, here are five things I learned in the process of making this project!

1) Automating basic tasks is not overkill

This seems kind of obvious, but while writing Cypress tests and using TravisCI to validate my small app's code is overkill, using TravisCI to automate my deploys to Github Pages was definitely not. It saved a lot of repetitive work merging pull requests, then pulling those changes locally so that I can run my deploy script manually. Troubleshooting config issues might not be nearly as fun, but being able to "set it once and forget it" gives me time to work on other things and I'm definitely doing this for my future projects.

And just as a quick illustration, I've added a small graphic overviewing both workflows:

Open Pull Request => Validate/ merge changes => Manually pull changes locally => Manually run deploy script => Repeat

Configure CI/ Troubleshoot CI config => Open Pull Request => Validate/ merge Changes => CI handles deploy => Repeat

2) Cypress and Jest would go great together if I made tests for both

I learned about Cypress as an end To end testing framework(see this Kent C Dodds post if you're not sure what that is) a while ago and wanted to check it out for this project. Previously I used a combination of Jest and React Testing Library in my last project in order to weirdly handle unit tests, integration tests, and even end to end tests.

That wasn't my most favorite thing in the world and as you can probably tell. Some of the tests for full app layouts probably belong in a series of end to end tests rather than leaving them in Jest.

In the context of this app, the end to end tests were simple, but were very easy to implement in Cypress. There's not a lot of complicated functionality in the app, but being able to validate about 85% of the app with Cypress is very powerful. If I were to increasescale the complexity of the app, then I'm confident that I'd end up making tests with both Jest and Cypress, where Jest tests can focus on functionality of components, and Cypress tests can focus on functionality of the whole app.

3) React.useEffect is about synchronizing state, not lifecycle

Every once and a while you need to relearn or rethink something you thought you were familiar with. The useEffect() hook was that for me during this project. I learned React before React Hooks was released, so I learned about class components and the component lifecycle. So, when I learned React Hooks, and the useEffect() hook, I think I ended up thinking that useEffect() was like componentDidMount() and such.

The useEffect() hook is not that, and through Dan Abramov's article on React.useEffect and Hooks, I had to relearn how that hook worked.

The short version is that useEffect() is more about data synchronization rather than lifecycle. So it's not that the hook goes through componentWillMount, componentShouldUpdate, but it's more that it's checking for changes, and then performing that effect if state and props in the dependency array chance.

This gave me the chance to rework some wonky data flow and synchronization as a result of relearning previous mental models. In the end, I ended up dumping some unnecessary local state and made the component update values at very set times.

4) Caching data is great

I also took the time to learn to use Redis in this app. Granted there isn't a lot of caching going on, but being able to cache a dataset of 3000+ entries is pretty nice. If I wanted to go further, I could have converted the React app into an isomorphic app and take advantage of caching the entire generated markup. Scaling it up, if there was user authentication like for my last project, I could have cached the markup for authenticated users and send that to their browsers when requested for even faster interactivity times.

5) D3 is not a data visualization library

D3 is not a data visualization library. This is kind of straight-forwards, but D3 doesn't not directly manipulate HTML elements. It does however, bind data to nodes in a tree and then decides what to render to the user based on the data. This is based on a Mike Bostock article on how D3 selections work and it's super important because this single principle is what D3 is built around.

It lets D3 be super flexible since you can programmatically adjust your markup as your data changes, but it comes at the cost of complexity. Understanding how D3 selects and binds data is super important and makes D3 much more powerful as a library as you can pretty much build any HTML you want with your data.

Fin

And that's all! This list isn't meant to be super comprehensive, but is just a highlight of some of the things that I thought might be useful to share.

You can check out the finished project there. The app is very very simple, but I enjoyed working on it and learned a ton of stuff from it!

Dockerizing a Python app

Wilson — Wed, 15 Apr 2020 20:48:48 +0000

Overview

Python's one of the more popular programming languages with a wide variety of applications, from building backends to creating a neural network. However, keeping your Python environment consistent between projects can be a pain as seen from the above.

This is where Docker comes in. Docker allows for containerization of applications, which means you can run isolated instances of your services and applications. In addition, Docker gives you the ability to connect your apps and services together with Docker Compose, which makes Python apps very convenient to build.

Note that all the code referenced in this article comes from this Github repo. This is a live app, and while the code in that particular release is not production ready, it still can run everything through Docker.

What's Docker?

What is Docker anyways? I went over it briefly above, and as mentioned, it's software that lets you containerize apps and services. The benefit of it is that instead of installing and using multiple versions of Python and managing multiple virtual environments, you can have Docker build each app separately with their own isolated environments.

This means we don't need to run python -m venv venv every time we build a new project, so as long as we setup a DockerFile and a docker-compose.yml file.

Images, Containers and Composing

Cool so with that out of the way, we need to overview some quick vocab.

First is Docker images. An image in Docker, is basically an immutable "snapshot" of an app. It includes instructions to start the app in a new container, and can be built using other images. The immutability is important as it allows us to have multiple versions of the same app, without having the version of the image collide.

Second is Docker containers. A Docker container is the instance of the app that we run based off of our image. It's analogous to a reproduction of the aforementioned "snapshot" and cannot be shared. We can however, spin up as many Docker containers of a particular image as we want, which is super handy for databases and similar services!

Installing Docker

If you don't have it already, you'll need to install Docker Desktop. If you already have it and know how to go through it, then you can skip to the next section.

I'll also briefly go over some of the commands you can run with the Docker CLI.

  docker image ls

docker image ls lists all of the images currently available on your computer.

  docker container ls

docker container ls lists all currently started/ active containers.

  docker container ls -a

docker container ls -a lists all containers whether they've been started or if they're active. The -a flag is short for --all

  docker container rm <first 3 symbols of the container ID or container name> [...other containers you want to remove]

docker container rm removes the listed containers. Note that you need either the first 3 symbols of the contaienr ID/ the container name AND the container can't be running before you remove a container.

  docker [COMMAND] --help

The help flag is super helpful and can be used with any command. It'll give you a list of options and also general instructions on how to use the commands.

Setting Up the Docker Env

With that said, to set up a Docker env, we need a couple of things.

we need to make an image of our app for Docker to use
we need to somehow coordinate all of the databases that our app uses

For the first point, that's what a Dockerfile is for. A Dockerfile is a file that defines the command line arguments required to set up and run the app. In English, it's a file that specifies the bare minimum required to set up and run the app.

Below is an example of a Dockerfile:

FROM python:3.8

WORKDIR /src

COPY ./requirements.txt .

RUN pip install -r requirements.txt

COPY . .

EXPOSE 5000

CMD ["flask", "run", "--host", "0.0.0.0"]

What's in the Dockerfile

So a quick overview of what everything is:

The first line specifies that we're basing our app's image from the Python 3.8 image

since we can build our image off of other images, we're taking full advantage of that
additional images can be found in DockerHub
we specify what tag/ version we want with a colon followed with the tag, so here python:3.8 specifies Python at the 3.8 tag

The second line sets up the working directory for the container. This is important as that'll be where our app's files will live.

The third line copies the requirements.txt file into our working directory. We could copy everything all at once but there's actually an important reason we don't that'll be explained next.

The fourth line runs whatever command we listed. So here it's installing our Python packages from the requirements.txt file.

we do this separately as Docker caches any external packages that we might need, depending on changes in requirements.txt(or package.json for NPM)
if we didn't do that, then Docker would just end up dumping the cached packages everytime, wasting a lot of time that could have been saved

The fifth line copies the rest of the files from our app's directory into the working directory. If we wanted to, we could also add it to a different sub-directory and it'd look like so:

COPY . ./my-subdirectory

The sixth line exposes a port for us to access the container. So it'd be port 5000 here.

The last line states the default command to execute on container start. Note that every part of the command is in an array.

Note that this array of strings is the preferred way to do it, and that we must wrap each command fragment needs to be wrapped in double quotes(") as it's parsed as a JSON array

Docker Compose

Cool, so what about databases. Sure we can run the database locally, but since Docker also has database images, we can actually run as many containers with our database on it as we want.

There's a big problem though, we don't want to manually create and start every single service that our app depends on. Sure it's fine if we have one database, but what we have two different databases and another app/ service that needs it's own database?

This is where docker-compose comes in(and Kubernetes as you scale up, but this is a different topic). Rather than having to manually start and create containers of the services your app depends on, you can use a docker-compose.yml file to automate large portions of that.

Check out the following link to an example docker-compose.yml file.

Sadly this won't be a full guide on making a Docker Compose file, but I'll overview the parts.

So in the file we have the below(plus or minus some comments):

version: '3.5' # Docker compose version
services:
  app:
    build:
      context: .
    # Binding the current working directory to the new container
    volumes:
      - .:/
    # The working directory in the app Dockerfile
    working_dir: ''
    command: ''
    links:
      - app-redis
    # ENV configuration
    env_file: .env
    # Additional app secrets
    environment:
      APP_TOKEN: /run/secrets/app_token
    secrets:
      - app_token
    ports:
      - '5000:5000'
  app-redis:
    image: redis:5
    ports:
      - '6379:6379'
# App secret registration
secrets:
  app_token:
    file: app_token.txt

Broadly speaking there are three major parts to the compose file:

the version of Docker Compose we want
- specifying the version of Docker Compose is important as certain properties aren't available in other versions, check the Docker Compose v3 docs for more
registering services
registering container secrets

Within the section for registering our services, we have the below:

configuring our Python app container
- linking our container to another service(Redis)
- binding our working directory so that we can live update the container
- adding an .env file
- adding our app secret
- exposing ports and start command
configuring out Redis container
- adding the image to build the container from
- exposing ports

Note that the links property is there to explicitly say that the app-redis service is a dependency for our Python app. It means that we access our database differently in our Flask app than normal but we'll get to that.

Required Updates to Our Python App

Now that we have out docker-compose.yml file up, we can't access our Redis instance the same way we might think. So rather than using localhost to have our Python app access our Redis instance, we need to update it so that it uses the service name instead like so:

# Fetch.py
# Std lib imports
import os
import json
from pathlib import Path

## Lib Imports
import redis
import requests
from dotenv import load_dotenv

# Global Module Var
isDocker = os.getenv('IS_DOCKER')

cacheHost = 'app-redis'
# ...Extra code
red_cache = redis.Redis(host = cacheHost, port = 6379)
# ...Rest of the module's implementation

Pretty quick and easy, and with that we can run docker-compose up inside our app directory and start the Flask app.

Conclusion

So we summarized how to quickly Dockerize a Python app. This involved the below:

creating a Dockerfile with:
- the image we're basing the app off of
- the contents we're copying
- any terminal commands we want to run
- a startup command
- a port to expose
creating a docker-compose.yml file which had:
- the version of Docker Compose
- a list of the services that will be created and run
- said services will need to specify if they're linked to one another
- in addition, the service with our Python app had additional .env and secret setup
- any secrets we wanted in our app

The end result is an app whose dependencies are fulfilled by Docker rather than the local machine. This results in a development environemnt that performs and is set up consistently every single time.

How I Upgraded My Portfolio

Wilson — Fri, 06 Mar 2020 22:00:55 +0000

Overview

Hey, so if you're reading this, that means my new portfolio is up in some way shape or form. It's light on content for now, but this is wildly preferable to what I had before. This article is a short run down on the upgrade process.

What Was v2.0.0?

Before we can get into the nitty gritty of how I upgraded my portfolio, I feel it's important to go over the old portfolio. The old portfolio was a static site built with plain HTML and Sass hosted on Github Pages. I made it mobile first, and I tried to make the project highlights as easy and quick of a browsing experience as possible.

Design-wise, I had a full research, mockup, iterate process and the end result, while simple, was pretty good.

Since it was entirely static assets, it was quite a lot of work to update the portfolio with new content or pages so that fell off very quickly.

v3.0 Requirements

v2.0 was pretty nice at the time, but it's been quite literally 2 years since I made it so some upgrades were necessary. At the moment, I'm very comfortable with React, and I've started to get into using Vue, so I wanted to use one of the two to build my portfolio. In addition, I wanted to be able to easily add new content into the site without having to manually generate new pages.

Deciding between Vue and React was actually pretty easy. I knew my app was going to be very simple, and require very little state to hang around. That kind of disqualifies React as writing JSX feels more involved than writing with Vue's templating syntax.

As an aside, the choice between the two isn't actually that important. Both frameworks would have done the job fine, and choosing one framework means you're also taking on the downsides of that framework.

In this situation, it means amongst other things, having a smaller community to find help in, more rigid rendering rules, and the framework having a potentially shaky future since it was developed outside of a company.

Basically, don't get hung on what framework is better, use the tool that's right for the job, and don't get dogmatic about someone's tool choices.

As for how I handled content, there are a multitiude of options(CMS, Headless CMS, etc), but the one I wanted to check out was static site generation. Static site generation(or generators) lets you use React or Vue to build your pages and components, while also letting you build pages from your static content and data. In the end the static site generator builds all of the required pages from the data and from the templates you made. Very slick!

Vue's static site generation framework is Gridsome. It uses GraphQL to query your data, and while it's super young, it's still solid. It also features a growing plugin library that can handle a variety of things from finding and including files into GraphQL, to transforming Markdown into your Vue pages.

Designing The Layout

Next up was designing a layout that would present everything I wanted in a user-friendly way. The current layout wasn't bad, but I felt that it had a bit too much going on in comparison to what I wanted. So went and did some research on web dev portfolios again with an emphasis on simplistic layouts and designs.

I ended up with the below list of portfolios:

The above portfolios, while simplistic, were all stylistically unique. Imitating that would be difficult since I don't have nearly as much content to put on there, but is still worth doing. I ended up doing a layout that was a wombo combo of the three in some way, which you can find on my portfolio.

Just like the v2.0 portfolio, it features cards for projects, and a short blurb, but the big difference is there's a dedicated About page and Projects page. In addition, some pages you can't actively navigate to right now as there isn't enough content to justify it.

Coding

The actual coding part of this was pretty wicked fast once I felt comfortable with what I planned to do. In comparison to my first portfolio, which took a month, this took about 12 days. There was some configuration stuff I had to mess with and several layout changes along the way, but other than that the development process was pretty fluid.

Writing Vue templates felt faster than writing a React component in a lot of ways, and conditional rendering feels way nicer. In addition, the simple layout meant very simple styling, so most of the styling didn't grow long enough to be extracted. Not to mention, that there was very little JavaScript that needed to be written, which meant less reasoning about what's changing and what needs to render and etc.

I did hit a small roadbump in querying for my data via GraphQL, but the bundled GraphQL GUI is super handy for testing and there's very minimal special Gridsome syntax to make sure we can query our data. Getting past that was not a huge issue and the GUI basically made everything possible.

What's Next

The portfolio's "done", but there's some extra tasks to do. First and foremost, I need to add more content; projects-wise, 3 is probably fine to start, but only having one blog article is not great. I also need to add in the Blogs page and the Contact page, with the contact page being a bit lower priority(unless I go freelancing).

Other than that, there's rolling tasks like the below:

fix wonky responsiveness
update sections with better designs as I get better at design
update the code to keep up with future releases(Vue, Gridsome, etc)
add extra pages for other hobbies and the like
pagination!

Obviously the list isn't comprehensive, and there's definitely going to be more that will be done as time goes on.