DEV Community: Daniel Starner

Serve Static Internal Documentation Behind OAuth Authentication

Daniel Starner — Sat, 02 Jul 2022 04:31:33 +0000

Background

Engineering teams frequently have centralized documentation websites that tie together all the different teams, projects, and workflows supported internally by the company. I have spoken about the importance of quality documentation before, and why its important to present internal documentation in an easy-to-read, easy-to-search manner.

Maintaining Quality Documentation

Daniel Starner ・ Sep 9 '21

#codequality #productivity #development #ux

In that post, I talked about how my team ended up writing our documentation using Docusaurus, a React- & Markdown-driven static site generator. It suited our needs well, looked great, and was very easy to write customized documentation and components using the MDX features.

We hosted this documentation site internally, with no authentication required since viewing it needed a user to be on the company VPN already.

I wanted to bring this same documentation style over to other companies, but they did not enforce the same VPN restrictions as Bloomberg. With this constraint, we needed a way to force authentication to our Docusaurus site before showing content to users. Authentication & private content was already discussed a few times on the project repository, but was ultimately rejected since that is outside the scope of a static-site generator.

So what can we do?

Solution

Let's walk through how to deploy Docusaurus behind an OAuth proxy which will force users to log in with a 3^rd party provider before viewing our documentation.

Tools & Concepts

Our documentation will be generated using the Docusaurus static-site generator
The site will be protected using GitHub sign-in by fronting it with Oauth2 Proxy
The site will be hosted on Heroku for ease, but you can change this for your specific project.

What is Oauth?

Oauth (Open Authorization) is an authorization protocol that allows a user to authenticate and access one service by allowing another service to provide your basic account details. OAuth allows for password-less logins - which are inherently safer - and requires users to maintain fewer accounts & credentials across services.

An example of Oauth is when a website allows you to log in with your Google, Twitter, or GitHub credentials. The site has preconfigured routes, configurations, and protocols to interact with those 3^rd party providers, so you can easily log in without needing credentials.

Our example will allow users to log in with their GitHub account to view the documentation.

The Process

This was performed on Docusaurus 2.0.0-beta.21

With some background on what we are doing, let's dive into setting up the project. This will walk through the individual steps, but you can view the complete starter project if you want to just get something deployed.

dstarner / oauth-gated-docusaurus

Running Docusaurus gated behind an OAuth provider session

Creating the Docusaurus Project

To start, we will create a basic Docusaurus project. This guide assumes you already have Node 16.x and npm installed on your machine, but you can use the nvm tool to configure these if needed.

Create a new Docusaurus site with the classic (default) template.

npx create-docusaurus@latest oauth-gated-docusaurus classic

Assume that the rest of the steps are run inside this newly created oauth-gated-docusaurus directory.

Docs-Only Mode

Since this will be internal documentation only, I usually delete the blog & homepage and clean up the docusaurus.config.js to match.

rm -rf blog src/pages src/components/*

Next, replace the presets key in the docusaurus.config.js file with the following:

presets: [
    [
      'classic',
      /** @type {import('@docusaurus/preset-classic').Options} */
      ({
        docs: {
          sidebarPath: require.resolve('./sidebars.js'),
          routeBasePath: '/', // Serve the docs at the site's root
          // Please change this to your repo.
          // Remove this to remove the "edit this page" links.
          editUrl:
            'https://github.com/facebook/docusaurus/tree/main/packages/create-docusaurus/templates/shared/',
        },
        blog: false,
        theme: {
          customCss: require.resolve('./src/css/custom.css'),
        },
      }),
    ],
  ],

Make sure to update the GitHub URL(s) to point to your repository instead.

Finally, create a landing page for your blog site. The slug: / denotes this file as the root URL. It is the only file that should have the slug key defined.

cat <<EOT >> docs/index.md
---
sidebar_position: 1
sidebar_label: Homepage
slug: /
---

# Welcome to Our Documentation

Hello world!

EOT

At this point, you should be able to npm start the server and open up the URL Docusaurus shows in the immediate output.

If you try to run npm run build, you may encounter a few build errors due to invalid URLs. This is left to an exercise for the user, but the example repository already has the fixes. It comes down to removing the to: blog and replacing /docs/intro with /intro to get the project to build nicely.

If you do not want to fix these issues, you can also set onBrokenLinks: 'warn' if you like. See the onBrokenLinks documentation here.

Deploy Basic Version to Heroku

Before adding in the authentication and authorization components, we will get the basic site up on Heroku, so we have a URL to redirect back to when configuring our OAuth application later.

Create the Heroku app. Create a Heroku account and install the Heroku CLI as needed.
```
heroku create
```
Add the nodejs buildpack which will run npm build automatically before deployment
```
heroku buildpacks:add heroku/nodejs
```

Add a Procfile to run Docusaurus in a web process

cat <<EOT >> Procfile
web: npm run serve -- -p \$PORT
EOT

Commit the changes and push them to GitHub & Heroku

git add . && git commit -m "prepare for heroku deployment"
git push heroku main && git push origin main

With the following command, you should be able to view your documentation in the browser, but there's one issue...it's still public! So let's fix that with some OAuth and a proxy.

heroku open

Creating the GitHub OAuth Credentials

With Docusaurus configured and running locally, we can begin creating an OAuth application. We will be using GitHub as an application provider, meaning users will log in via it, redirecting them back to our documentation. I chose GitHub because I assume most people reading this article have a GitHub account, and any GitHub user can create OAuth applications without paying.

Visit the page to register a new app and provide the required details.
1. Application Name: Docusaurus OAuth Example
2. Homepage URL: Whatever heroku open opened in your browser
3. Authorization callback URL: <Homepage URL >/oauth2/callback

Once the OAuth application is registered, note the Client ID, which should be a random-looking string of 16-24 characters. Save this to Heroku to be used later with the following command.

heroku config:set OAUTH2_PROXY_CLIENT_ID=<value from GitHub>

Generate a Client Secret

Finally, you will need to generate a client secret for the OAuth application. Under the Client secrets section, click the Generate a new client secret button, which will reload the page with a newly minted client secret key. Make sure to copy it manually or by clicking the Copy icon because you cannot retrieve this key again!

Save this to the Heroku app to be used later.

heroku config:set OAUTH2_PROXY_CLIENT_SECRET=<value from GitHub>

Adding OAuth Proxy to Docusaurus

Finally, all of the pieces are in place to run our documentation site behind an OAuth2 Proxy.

A Proxy is a middleman between users trying to access the website and the web process hosting the content. Its job is to either show the website content to authorized users or redirect those not authorized to GitHub.

Begin by adding the heroku-buildpack-oauth2-proxy community buildpack to your application.

heroku buildpacks:add cfra/oauth2-proxy

We added the GitHub OAuth client and secret keys in the last step, but we need to tell our proxy to use GitHub for authentication. We can do this with the following:

heroku config:set OAUTH2_PROXY_PROVIDER=github

The last step is to generate a secret key to sign all authentication cookies. This string can be random, so we are using Python for this step. You may need to switch python for python3 in the command if you get a Python error.

heroku config:set OAUTH2_PROXY_COOKIE_SECRET=$(python -c \
    'from secrets import token_urlsafe; print(token_urlsafe(32)[:32])' \
)

To see the change, you will need to place the oauth2-proxy script in front of our npm serve ... command from earlier. Then, update the Procfile to the following.

web: /app/bin/start_with_oauth2_proxy.sh npm run serve -- -p 8080

Notice how the server is now running on 8080 instead of looking at the $PORT variable. This change is because oauth2_proxy runs listening on $PORT, and it expects our gated webserver to listen on 8080.

Testing Login

View your documentation in the browser again, and you should be greeted with the OAuth2 Proxy landing page. Try to sign in with GitHub now!

If all worked, you should be able to view your documentation again! We have successfully gated our documentation behind an OAuth Provider.

Suppose you would like to change the OAuth provider being used. In that case, you will need to update the OAUTH2_PROXY_PROVIDER, OAUTH2_PROXY_CLIENT_ID, and the OAUTH2_PROXY_CLIENT_SECRET variables on the app after reading the oauth2-proxy documentation for the associated provider type.

Bonus: Restricting to a GitHub Organization

Cool, our documentation now has OAuth in front of it, meaning people have to log into their GitHub accounts before viewing it. That's great, but we said this was for internal company documentation; we don't want just any GitHub user seeing our documentation.

Reading through the oauth2-proxy GitHub Provider documentation, we can see that the provider can implement more strict access controls at the organization, team, repository, token, or user level. Let's restrict this to a single team by adding a configuration argument.

heroku config:set OAUTH2_PROXY_ARGS="-github-org='my-cool-org'"

And update the Procfile to read the OAUTH2_PROXY_ARGS configuration variable.

web: /app/bin/start_with_oauth2_proxy.sh $OAUTH2_PROXY_ARGS npm run serve -- -p 8080

This means that changing who can access the documentation at different abstraction levels boils down to just updating the OAUTH2_PROXY_ARGS configuration variable, which can be done via the heroku CLI or dashboard under the app settings. Tinkering with this config var is left as a final exercise to the user.

Conclusion

With this setup, engineering (and non-engineering teams too!) can write easy-to-manage, easier-to-view documentation sites powered by Docusaurus while ensuring that only appropriate individuals can view them. This detail is usually critical for internal company documentation. Much more configuration and OAuth settings can be included by reading through the oauth2-proxy documentation.

Thank you for reading this post! I hope it helps you and your teams as much as figuring out how to make this work helped mine.

Managing Django Media & Static Files on Heroku with Bucketeer

Daniel Starner — Thu, 13 Jan 2022 18:22:54 +0000

This article will walk through how we correctly persist static & media files for a Django application hosted on Heroku. As a bonus, it will also explain how we can satisfy the additional constraint of specifying private versus public media files based on model definitions.

Before I begin, this post extends from this TestDriven.io article that was written awhile back. I frequent it often when setting up my projects, and have built some extra functionality on top of it over the years. I decided to create a more focused post that references Heroku & Bucketeer with these extra features after helping an individual on StackOverflow.

Dec 26 '21

I think it's because I turn off a PC, where I took these images

This probably is not it, because Heroku doesn't have access to the files on your computer.

When you upload a file to the Django admin, it looks at the DEFAULT_FILE_STORAGE settings configuration to determine how to…

Open Full Answer

So without further ado, let's first dive into what static & media files are and how Heroku dynos manage their filesystem?

What are Media & Static Files

If you are working with a Django project, then you inevitably have all of your Python application code written around a bunch of .py files. These are the code paths of your application, and the end-user - hopefully - never actually sees these files or their contents.

Outside of these business-logic files, it is common to serve users directly from your server's file system. For these static files, Django doesn't need to run any code for them; the framework looks up the file and returns the contents for the requesting user to view.

Some examples of static files include:

Non-templated HTML
CSS & JavaScript files to make your page look nice
User profile pictures
Generated PDFs

Media files in Django are a particular variant of static files. Media files are read from the server's file system as well. Unlike static files, though, they are usually generated files uploaded by users or generated by your application and are associated with a model's FileField or ImageField. In the examples above, user profile pictures and generated PDFs are typical examples of media files.

Django with Media & Static Files

When a new media file is uploaded to a Django web application, the framework looks at the DEFAULT_FILE_STORAGE settings configuration to determine how to store that file. By default, it uses the django.core.files.storage.FileSystemStorage class, which is what most projects start off as having configured. This implementation looks at the MEDIA_ROOT configuration that is defined in the settings.py file and copies the uploaded file contents to a deterministically-created file path under that given MEDIA_ROOT.

For example, if the MEDIA_ROOT is set as /var/www/media, all uploaded files will be copied and written to a location under /var/www/media/.

Heroku with Media & Static Files

Storing these static files on your server's disk file system is okay until you start to work with a containerization platform such as Heroku. To explain why this is the case, it helps to take a step back.

When downloading files on your personal computer, it's okay that these get written to the file system - usually under ~/Downloads or somewhere similar. This download is because you expect your computer's file system to persist across restarts and shutdowns; if you download a file and restart your computer, that downloaded file should still be there once the laptop is finished restarting.

Heroku uses containerization to execute customer workloads. One fact of this environment is that the associated file systems do not persist across restarts and reschedules. Heroku dynos are ephemeral, and they can be destroyed, restarted, and moved without any warning, which replaces the associated filesystem. This situation means that any uploaded files referenced by FileField's andImageField's are just deleted without a trace every time the dyno is restarted, moved, or scaled.

Complete Example Codebase

I will be stepping through the process of configuring the Django application for Heroku & S3-compatible storage, but feel free to reference the repository below for the complete code to browse through.

dstarner / django-heroku-static-file-example

Used in my blog post of detailing private & public static files for a Heroku-served Django application

Properly Managing Django Media & Static Files on Heroku Example

Used in my blog post of detailing private & public static files for a Heroku-served Django application.

Note: This does include a $5.00 / month Bucketeer add-on as a part of the one-click deployment.

View on GitHub

Bootstrapping Django on Heroku

This tutorial aims to help you retrofit an existing Django project with S3-compatible storage, but I'll quickly go through the steps I used to set up the example Django application. It may help those new to Django & Heroku or those who encounter bugs following the rest of the setup process.

You can view the tagged project before the storage change at commit 299bbe2.

Bootstrapped a Django project example
- Uses poetry for dependency management
- All of the Django code is under the example package, and the manage.py file is in the root. I've always found this structure cleaner than the Django apps defined in the project root.
Configured the project for Heroku
- django-heroku package to automatically configure ALLOWED_HOSTS, DATABASE_URL, and more. This reduces the headache of deploying Django on Heroku considerably
- A Procfile that runs a gunicorn process for managing the WSGI application
- An app.json is defined with some fundamental configuration values and resources defined for the project to work
- A release process definition in the Procfile and an associated scripts/release.sh script that runs staticfile collection and database migrations

Introducing Heroku's Bucketeer Add-On

Before we can start managing static and media files, the Django application needs a persistent place to store the files. Again, we can look to Heroku's extensive list of Add-Ons for s3-compatible storage. Ours of choice will be one called Bucketeer.

Heroku's Bucketeer add-on provides an AWS S3 storage bucket to upload and download files for our application. The Django application will use this configured bucket to store files uploaded by the server and download them from the S3 when a user requests the files.

If you'd like to learn more about AWS S3, the widely-popular data storage solution that Bucketeer is built upon, you can read the S3 user documentation.

It is worth mentioning that the base plan for Bucketeer - Hobbyist - is $5 per month. If you plan on spinning up the one-click example posted above, it should only cost a few cents if you proactively destroy the application when you are done using it.

Including the Bucketeer Add-On

To include the Bucketeer add-on in our application, we can configure it through the Heroku CLI, web dashboard, or via the project's app.json file. We will use the third method of including the add-on in an app.json file.

If the project does not have one already, we can create the basic structure listed below, with the critical part being the addition of the "add-ons" configuration. This array defines the "bucketeer:hobbyist" resource that our application will use, and Heroku will install the add-on into our application if it does not already exist. We also include the " as" keyword, which will preface the associated configuration variables with the term BUCKETEER. This prefacing is helpful to keep the generated configuration value names deterministic because, by default, Heroku will generate the prefix as a random color.



{
    // ... rest above
    "addons": [
        // ...other addons...
        {
            "plan": "bucketeer:hobbyist",
            "as": "BUCKETEER"
        }
    ]
}

With the required resources being defined, we can start integrating with our storage add-on.

Implementing Our Storage Solution

The django-storages package is a collection of custom, reuseable storage backends for Django. It aids immensely in saving static and media files to different cloud & storage provider options. One of the supported storage providers is S3, which our Bucketeer add-on is built on. We will leverage the S3 django-storages backend to handle different file types.

Installing `django-storages`

Begin by installing the django-storages package and the related boto3 package used to interface with AWS's S3. We will also lock our dependencies to ensure poetry and our Heroku deployment continue to work as expected.



poetry add django-storages boto3 && poetry lock

Then, just like most Django-related packages, django-storages will need to be added to the project's INSTALLED_APPS in the projects settings.py file. This will allow Django to load the appropriate code flows as the application starts up.



# example/config/settings.py
INSTALLED_APPS = [
    # ... django.X.Y apps above
    'storages',
    # ... custom project apps below
]

Implementing Static, Public & Private Storage Backends

We will return to the settings.py file later to configure the usage of django-storages, but before that can be done, we will implement three custom storage backends:

A storage backend for static files - CSS, Javascript, and publicly accessible images - that will be stored in version control - aka git - and shipped with the application
A public storage backend for dynamic media files that are not stored in version control, such as uploaded files and attachments
A private storage backend for dynamic media files that are not stored in the version control that require extra access to be viewed, such as per-user reports and potentially profile images. Files managed by this backend require an access key and will block access to those without a valid key.

We can extend from django-storages 's S3Boto3Storage storage backend to create these. The following code can be directly "copy and paste "'d into your project. The different settings attributes read in the module will be written shortly, so do not expect this code to work if you import it right now.



# FILE: example/utils/storage_backends.py

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    """Used to manage static files for the web server"""
    location = settings.STATIC_LOCATION
    default_acl = settings.STATIC_DEFAULT_ACL


class PublicMediaStorage(S3Boto3Storage):
    """Used to store & serve dynamic media files with no access expiration"""
    location = settings.PUBLIC_MEDIA_LOCATION
    default_acl = settings.PUBLIC_MEDIA_DEFAULT_ACL
    file_overwrite = False


class PrivateMediaStorage(S3Boto3Storage):
    """
    Used to store & serve dynamic media files using access keys
    and short-lived expirations to ensure more privacy control
    """
    location = settings.PRIVATE_MEDIA_LOCATION
    default_acl = settings.PRIVATE_MEDIA_DEFAULT_ACL
    file_overwrite = False
    custom_domain = False

The attributes listed in each storage backend class perform the following:

location: This dictates the parent directory used in the S3 bucket for associated files. This is concatenated with the generated path provided by a FileField or ImageField 's upload_to method.
default_acl: This dictates the access policy required for reading the files. This dictates the storage backend's access control through values of None, public-read, and private. django-storages and the S3Boto3Storage parent class with translate these into object policies.
file_overwrite: In most cases, it's better not to overwrite existing files if we update a specific path. With this set to False, a unique suffix will be appended to the path to prevent naming collisions.
custom_domain: Disabled here, but you can enable it if you want to use AWS's CloudFront and django-storage to serve from it.

Configure Settings to Use the Storage Backends

With our storage backends defined, we can configure them to be used in different situations via the settings.py file. However, it is challenging to use S3 and these different cloud storage backends while in development, and I've always been a proponent of keeping all resources and files "local" to the development machine, so we will create a logic path that will:

Use the local filesystem to store static and media files for convenience. The Django server will be responsible for serving these files directly.
Use the custom S3 storage backends when an environment variable is enabled. We will use the S3_ENABLED variable to control this, enabling it in our Heroku configuration variables.

First, we will assume that you have a relatively vanilla settings.py file concerning the static- & media-related variables. For reference, a new project should have a block that looks similar to the following:



# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.0/howto/static-files/

STATIC_URL = 'static/'

STATIC_ROOT = BASE_DIR / 'collected-static'

We will design a slightly advanced control flow that will seamlessly handle the two cases defined above. In addition, it will provide enough control to override each part of the configuration as needed.

Since there are already default values for the static file usage, we can add default values for media file usage. These will be used when serving files locally from the server while in development mode.



STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'collected-static'

MEDIA_URL = '/media/'
MEDIA_ROOT = BASE_DIR / 'collected-media'

To begin the process of including S3, let's create the controls to manage if we should serve static & media files from the local server or through the S3 storage backend. We will create three variables

S3_ENABLED: controls whether media & static files should use S3 storage by default
LOCAL_SERVE_MEDIA_FILES: controls whether media files should use S3 storage. Defaults to the negated S3_ENABLED value
LOCAL_SERVE_STATIC_FILES: controls whether static files should use S3 storage. Defaults to the negated S3_ENABLED value



from decouple import config  # import explained below

# ...STATIC and MEDIA settings here...

# The following configs determine if files get served from the server or an S3 storage
S3_ENABLED = config('S3_ENABLED', cast=bool, default=False)
LOCAL_SERVE_MEDIA_FILES = config('LOCAL_SERVE_MEDIA_FILES', cast=bool, default=not S3_ENABLED)
LOCAL_SERVE_STATIC_FILES = config('LOCAL_SERVE_STATIC_FILES', cast=bool, default=not S3_ENABLED)

if (not LOCAL_SERVE_MEDIA_FILES or not LOCAL_SERVE_STATIC_FILES) and not S3_ENABLED:
    raise ValueError('S3_ENABLED must be true if either media or static files are not served locally')

In the example above, we are using the python-decouple package to make it easier to read and cast environment variables to Python variables. I highly recommend this package when working with settings.py configurations. We also include a value check to ensure consistency across these three variables. If all three variables are defined in the environment but conflict with one another, the program will throw an error.

We can now start configuring the different configuration variables required by our file storage backends based on those control variables' value(s). We begin by including some S3 configurations required whether we are serving static, media, or both types of files.



if S3_ENABLED:
    AWS_ACCESS_KEY_ID = config('BUCKETEER_AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = config('BUCKETEER_AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = config('BUCKETEER_BUCKET_NAME')
    AWS_S3_REGION_NAME = config('BUCKETEER_AWS_REGION')
    AWS_DEFAULT_ACL = None
    AWS_S3_SIGNATURE_VERSION = config('S3_SIGNATURE_VERSION', default='s3v4')
    AWS_S3_ENDPOINT_URL = f'https://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}

The above defines some of the variables required by the django-storages S3 backend and sets the values to environment configurations that are provided by the Bucketeer add-on. As previously mentioned, all of the add-on environment variables are prefixed with BUCKETEER_. The S3_SIGNATURE_VERSION environment variable is not required and most likely does not need to be included.

With the S3 configuration together, we can reference the LOCAL_SERVE_MEDIA_FILES and LOCAL_SERVE_STATIC_FILES control variables to override the default static and media file settings if they are desired to be served via S3.



if not LOCAL_SERVE_STATIC_FILES:
    STATIC_DEFAULT_ACL = 'public-read'
    STATIC_LOCATION = 'static'
    STATIC_URL = f'{AWS_S3_ENDPOINT_URL}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'example.utils.storage_backends.StaticStorage'

Notice the last line where STATICFILES_STORAGE is set to the custom Backend we created. That ensures it follows the location & ACL (Access Control List) policies that we configured initially. With this configuration, all static files will be placed under /static/ in the bucket, but feel free to update STATIC_LOCATION if desired.

We can configure a very similar situation for media files.



if not LOCAL_SERVE_MEDIA_FILES:
    PUBLIC_MEDIA_DEFAULT_ACL = 'public-read'
    PUBLIC_MEDIA_LOCATION = 'media/public'

    MEDIA_URL = f'{AWS_S3_ENDPOINT_URL}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'example.utils.storage_backends.PublicMediaStorage'

    PRIVATE_MEDIA_DEFAULT_ACL = 'private'
    PRIVATE_MEDIA_LOCATION = 'media/private'
    PRIVATE_FILE_STORAGE = 'example.utils.storage_backends.PrivateMediaStorage'

The big difference here is that we have configured two different storage backends for media files; one for publicly accessible objects and one for objects that require an access token. When the file is requested, this token will be generated internally by django-storages so you do not have to worry about anonymous public access.

Local Development Serving

Since we will have S3_ENABLED set to False in our local development environment, it will serve static and media files locally through the Django server instead of from S3. We will need to configure the URL routing to handle this scenario. We can configure our urls.py file to serve the appropriate files like so:



from django.conf import settings
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import path


urlpatterns = [
    path('admin/', admin.site.urls),
]

if settings.LOCAL_SERVE_STATIC_FILES:
    urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

if settings.LOCAL_SERVE_MEDIA_FILES:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

This will locally serve the static or media files based on the values of the LOCAL_SERVE_STATIC_FILES and LOCAL_SERVE_MEDIA_FILES settings variables we defined.

Enabling S3 Storage

We can enable these storages and our add-on in the app.json file to start using these storage backends. This will effectively disable LOCAL_SERVE_STATIC_FILES and LOCAL_SERVE_MEDIA_FILES to start serving both via S3 when deployed to Heroku.



{
  // ...rest of configs...
  "env": {
    // ...rest of envs...
    "S3_ENABLED": {
      "description": "Enable to upload & serve static and media files from S3",
      "value": "True"
    },
  }
}

Using the Private Storage

By default, Django will use the PublicMediaStorage class for uploading media files, meaning the contents will be publicly accessible to anyone with the link. However, a model can utilize the PrivateMediaStorage backend when desired, which will create short-lived access tokens that prevent the public from viewing the associated object.

The below is an example of using public and private media files on the same model.



from django.db import models

from example.utils.storage_backends import PrivateMediaStorage


class Organization(models.Model):
    """A sample Organization model with public and private file field usage
    """

    logo = models.ImageField(help_text='A publicly accessible company logo')

    expense_report = models.FileField(
        help_text='The private expense report requires a short-lived access token'
        storage=PrivateMediaStorage()  # will create private files
    )

You can see the code for this complete example at commit 265becc. This configuration will allow your project to scale efficiently using Django on Heroku using Bucketeer.

In a future post, we will discuss how to upload and set these files using vanilla Django & Django REST Framework.

As always, if you find any bugs, issues, or unclear explanations, please reach out to me so I can improve the tutorial & experience for future readers.

Take care everyone

Building Dynamic Breadcrumbs in NextJS

Daniel Starner — Wed, 05 Jan 2022 01:25:37 +0000

Breadcrumbs are a website navigation tool that allows users to see their current page's "stack" of how it is nested under any parent pages. Users can then jump back to a parent page by clicking the associated breadcrumb link. These "Crumbs" increase the User Experience of the application, making it easier for the users to navigate nested pages efficiently and effectively.

Breadcrumbs are popular enough while building a web dashboard or application that you may have considered adding them. Generating these breadcrumb links efficiently and with the appropriate context is key to an improved user experience.

Let's build an intelligent NextBreadcrumbs React component that will parse the current route and create a dynamic breadcrumbs display that can handle both static & dynamic routes efficiently.

My projects usually revolve around Nextjs and MUI (formerly Material-UI), so that is the angle that I am going to approach this problem from, although the solution should work for any Nextjs-related application.

Static Route Breadcrumbs

To begin, our NextBreadcrumbs component will only handle static routes, meaning that our project has only static pages defined in the pages directory.

The following are examples of static routes because they do not contain ['s and] 's in the route names, meaning the directory structure lines up 1:1 precisely with the expected URLs that they serve.

pages/index.js --> /
pages/about.js --> /about
pages/my/super/nested/route.js --> /my/super/nested/route

The solution will be extended to handle dynamic routes later.

Defining the Basic Component

We can start with the fundamental component that uses the MUI Breadcrumbs component as a baseline.



import Breadcrumbs from '@mui/material/Breadcrumbs';
import * as React from 'react';

export default function NextBreadcrumbs() {
  return (
    <Breadcrumbs aria-label="breadcrumb" />
  );
}

The above creates the basic structure of the NextBreadcrumbs React component, imports the correct dependencies, and renders an empty Breadcrumbs MUI component.

We can then add in the next/router hooks, which will allow us to build the breadcrumbs from the current route.

We also create a Crumb component that will be used to render each link. This is a pretty dumb component for now, except that it will render basic text instead of a link for the last breadcrumb.

In a situation like /settings/notifications, it would render as the following:



Home (/ link) > Settings (/settings link) > Notifications (no link)

The user is already on the last breadcrumb's page, so there is no need to link out to the same page. All the other crumbs are rendered as links to be clicked.



import Breadcrumbs from '@mui/material/Breadcrumbs';
import Link from '@mui/material/Link';
import Typography from '@mui/material/Typography';
import { useRouter } from 'next/router';
import React from 'react';


export default function NextBreadcrumbs() {
  // Gives us ability to load the current route details
  const router = useRouter();

  return (
    <Breadcrumbs aria-label="breadcrumb" />
  );
}


// Each individual "crumb" in the breadcrumbs list
function Crumb({ text, href, last=false }) {
  // The last crumb is rendered as normal text since we are already on the page
  if (last) {
    return <Typography color="text.primary">{text}</Typography>
  }

  // All other crumbs will be rendered as links that can be visited 
  return (
    <Link underline="hover" color="inherit" href={href}>
      {text}
    </Link>
  );
}

We can then dive back into the NextBreadcrumbs component to generate the breadcrumbs from the route with this layout. Some existing code will start to be omitted to keep the code pieces smaller. The full example is shown below.

We will generate a list of breadcrumb objects that contain the information to be rendered by each Crumb element. Each breadcrumb will be created by parsing the Nextjs router's asPath property, which is a string containing the route as shown in the browser URL bar.

We will strip any query parameters, such as ?query=value, from the URL to simplify the breadcrumb creation process.



export default function NextBreadcrumbs() {
  // Gives us ability to load the current route details
  const router = useRouter();

  function generateBreadcrumbs() {
    // Remove any query parameters, as those aren't included in breadcrumbs
    const asPathWithoutQuery = router.asPath.split("?")[0];

    // Break down the path between "/"s, removing empty entities
    // Ex:"/my/nested/path" --> ["my", "nested", "path"]
    const asPathNestedRoutes = asPathWithoutQuery.split("/")
                                                 .filter(v => v.length > 0);

    // Iterate over the list of nested route parts and build
    // a "crumb" object for each one.
    const crumblist = asPathNestedRoutes.map((subpath, idx) => {
      // We can get the partial nested route for the crumb
      // by joining together the path parts up to this point.
      const href = "/" + asPathNestedRoutes.slice(0, idx + 1).join("/");
      // The title will just be the route string for now
      const title = subpath;
      return { href, text }; 
    })

    // Add in a default "Home" crumb for the top-level
    return [{ href: "/", text: "Home" }, ...crumblist];
  }

  // Call the function to generate the breadcrumbs list
  const breadcrumbs = generateBreadcrumbs();

  return (
    <Breadcrumbs aria-label="breadcrumb" />
  );
}

With this list of breadcrumbs, we can now render them using the Breadcrumbs and Crumb components. As previously mentioned, only the return portion of our component is shown for brevity.



  // ...rest of NextBreadcrumbs component above...
  return (
    {/* The old breadcrumb ending with '/>' was converted into this */}
    <Breadcrumbs aria-label="breadcrumb">
      {/*
        Iterate through the crumbs, and render each individually.
        We "mark" the last crumb to not have a link.
      */}
      {breadcrumbs.map((crumb, idx) => (
        <Crumb {...crumb} key={idx} last={idx === breadcrumbs.length - 1} />
      ))}
    </Breadcrumbs>
  );

This should start generating some very basic - but working - breadcrumbs on our site once rendered; /user/settings/notifications would render as



Home > user > settings > notifications

Memoizing Generated Breadcrumbs

There is a quick improvement that we can make before going further, though. The breadcrumb list is recreated every time the component re-renders, so we can memoize the crumb list for a given route to save some performance. We can wrap our generateBreadcrumbs function call in the useMemo React hook.



  const router = useRouter();

  // this is the same "generateBreadcrumbs" function, but placed
  // inside a "useMemo" call that is dependent on "router.asPath"
  const breadcrumbs = React.useMemo(function generateBreadcrumbs() {
    const asPathWithoutQuery = router.asPath.split("?")[0];
    const asPathNestedRoutes = asPathWithoutQuery.split("/")
                                                 .filter(v => v.length > 0);

    const crumblist = asPathNestedRoutes.map((subpath, idx) => {
      const href = "/" + asPathNestedRoutes.slice(0, idx + 1).join("/");
      return { href, text: subpath }; 
    })

    return [{ href: "/", text: "Home" }, ...crumblist];
  }, [router.asPath]);

  return // ...rest below...

Improving Breadcrumb Text Display

Before we start incorporating dynamic routes, we can clean this current solution up more by including a nice way to change the text shown for each crumb generated.

Right now, if we have a path like /user/settings/notifications, then it will show:



Home > user > settings > notifications

...which is not very appealing. We can provide a function to the NextBreadcrumbs component to generate a more user-friendly name for each of these nested route crumbs.




const _defaultGetDefaultTextGenerator= path => path

export default function NextBreadcrumbs({ getDefaultTextGenerator=_defaultGetDefaultTextGenerator }) {
  const router = useRouter();

  // Two things of importance:
  // 1. The addition of getDefaultTextGenerator in the useMemo dependency list
  // 2. getDefaultTextGenerator is now being used for building the text property
  const breadcrumbs = React.useMemo(function generateBreadcrumbs() {
    const asPathWithoutQuery = router.asPath.split("?")[0];
    const asPathNestedRoutes = asPathWithoutQuery.split("/")
                                                 .filter(v => v.length > 0);

    const crumblist = asPathNestedRoutes.map((subpath, idx) => {
      const href = "/" + asPathNestedRoutes.slice(0, idx + 1).join("/");
      return { href, text: getDefaultTextGenerator(subpath, href) }; 
    })

    return [{ href: "/", text: "Home" }, ...crumblist];
  }, [router.asPath, getDefaultTextGenerator]);

  return ( // ...rest below

Then our parent component can have something like the following: to title-ize the subpaths, or maybe even replace them with a new string.



{/* Assume that `titleize` is written and works appropriately */}
<NextBreadcrumbs getDefaultTextGenerator={path => titleize(path)} />

This implementation would then result in the following breadcrumbs. The complete code example at the bottom has more examples of this.



Home > User > Settings > Notifications

Nextjs Dynamic Routes

Nextjs's router allows for including dynamic routes that uses Pattern Matching to enable the URLs to have slugs, UUIDs, and other dynamic values that will then be passed to your views.

For example, if your Nextjs application has a page component at pages/post/[post_id].js, then the routes /post/1 and /post/abc will match it.

For our breadcrumbs component, we would like to show the name of the associated post instead of just its UUID. This means that the component will need to dynamically look up the post data based on the nested URL route path and regenerate the text of the associated crumb.

Right now, if you visit /post/abc, you would see breadcrumbs that look like



post > abc

but if the post with UUID has a title of My First Post, then we want to change the breadcrumbs to say



post > My First Post

Let's dive into how that can happen using async functions.

Nextjs Router: `asPath` vs `pathname`

The next/router router instance in our code has two useful properties for our NextBreadcrumbs component; asPath and pathname. The router asPath is the URL path as shown directly in the browser's URL bar. The pathname is a more internal version of the URL that has the dynamic parts of the path replaced with their [parameter] components.

For example, consider the path /post/abc from above.

The asPath would be /post/abc as the URL is shown
The pathname would be /post/[post_id] as our pages directory dictates

We can use these two URL path variants to build a way to dynamically fetch information about the breadcrumb, so we can show more contextually appropriate information to the user.

There is a lot going on below, so please re-read it and the helpful notes below a few times over if needed.




const _defaultGetTextGenerator = (param, query) => null;
const _defaultGetDefaultTextGenerator = path => path;

// Pulled out the path part breakdown because its
// going to be used by both `asPath` and `pathname`
const generatePathParts = pathStr => {
  const pathWithoutQuery = pathStr.split("?")[0];
  return pathWithoutQuery.split("/")
                         .filter(v => v.length > 0);
}

export default function NextBreadcrumbs({
  getTextGenerator=_defaultGetTextGenerator,
  getDefaultTextGenerator=_defaultGetDefaultTextGenerator
}) {
  const router = useRouter();

  const breadcrumbs = React.useMemo(function generateBreadcrumbs() {
    const asPathNestedRoutes = generatePathParts(router.asPath);
    const pathnameNestedRoutes = generatePathParts(router.pathname);

    const crumblist = asPathNestedRoutes.map((subpath, idx) => {
      // Pull out and convert "[post_id]" into "post_id"
      const param = pathnameNestedRoutes[idx].replace("[", "").replace("]", "");

      const href = "/" + asPathNestedRoutes.slice(0, idx + 1).join("/");
      return {
        href, textGenerator: getTextGenerator(param, router.query),
        text: getDefaultTextGenerator(subpath, href)
      }; 
    })

    return [{ href: "/", text: "Home" }, ...crumblist];
  }, [router.asPath, router.pathname, router.query, getTextGenerator, getDefaultTextGenerator]);

  return ( // ...rest below

The asPath breakdown was moved to a generatePathParts function since the same logic is used for both router.asPath and router.pathname.
Determine the param'eter that lines up with the dynamic route value, soabcwould result inpost_id`.
The nested route param'eter and all associated query values (router.query) are passed to a provided getTextGenerator which will return either a null value or a Promise` response that should return the dynamic string to use in the associated breadcrumb.
The useMemo dependency array has more dependencies added; router.pathname, router.query, and getTextGenerator.

Finally, we need to update the Crumb component to use this textGenerator value if provided for the associated crumb object.



function Crumb({ text: defaultText, textGenerator, href, last=false }) {

  const [text, setText] = React.useState(defaultText);

  useEffect(async () => {
    // If `textGenerator` is nonexistent, then don't do anything
    if (!Boolean(textGenerator)) { return; }
    // Run the text generator and set the text again
    const finalText = await textGenerator();
    setText(finalText);
  }, [textGenerator]);

  if (last) {
    return <Typography color="text.primary">{text}</Typography>
  }

  return (
    <Link underline="hover" color="inherit" href={href}>
      {text}
    </Link>
  );
}

The breadcrumbs can now handle both static routes and dynamic routes cleanly, with the potential to display user-friendly values. While the above code is the component's business logic, this can all be used with a parent component that looks like the final example below.

Full Example



// NextBreadcrumbs.js

const _defaultGetTextGenerator = (param, query) => null;
const _defaultGetDefaultTextGenerator = path => path;

// Pulled out the path part breakdown because its
// going to be used by both `asPath` and `pathname`
const generatePathParts = pathStr => {
  const pathWithoutQuery = pathStr.split("?")[0];
  return pathWithoutQuery.split("/")
                         .filter(v => v.length > 0);
}

export default function NextBreadcrumbs({
  getTextGenerator=_defaultGetTextGenerator,
  getDefaultTextGenerator=_defaultGetDefaultTextGenerator
}) {
  const router = useRouter();

  const breadcrumbs = React.useMemo(function generateBreadcrumbs() {
    const asPathNestedRoutes = generatePathParts(router.asPath);
    const pathnameNestedRoutes = generatePathParts(router.pathname);

    const crumblist = asPathNestedRoutes.map((subpath, idx) => {
      // Pull out and convert "[post_id]" into "post_id"
      const param = pathnameNestedRoutes[idx].replace("[", "").replace("]", "");

      const href = "/" + asPathNestedRoutes.slice(0, idx + 1).join("/");
      return {
        href, textGenerator: getTextGenerator(param, router.query),
        text: getDefaultTextGenerator(subpath, href)
      }; 
    })

    return [{ href: "/", text: "Home" }, ...crumblist];
  }, [router.asPath, router.pathname, router.query, getTextGenerator, getDefaultTextGenerator]);

  return (
    <Breadcrumbs aria-label="breadcrumb">
      {breadcrumbs.map((crumb, idx) => (
        <Crumb {...crumb} key={idx} last={idx === breadcrumbs.length - 1} />
      ))}
    </Breadcrumbs>
  );
}


function Crumb({ text: defaultText, textGenerator, href, last=false }) {

  const [text, setText] = React.useState(defaultText);

  useEffect(async () => {
    // If `textGenerator` is nonexistent, then don't do anything
    if (!Boolean(textGenerator)) { return; }
    // Run the text generator and set the text again
    const finalText = await textGenerator();
    setText(finalText);
  }, [textGenerator]);

  if (last) {
    return <Typography color="text.primary">{text}</Typography>
  }

  return (
    <Link underline="hover" color="inherit" href={href}>
      {text}
    </Link>
  );
}

An example of this NextBreadcrumbs being used can be seen below. Note that useCallback is used to create only one reference to each helper function which will prevent unnecessary re-renders of the breadcrumbs when/if the page layout component re-rendered. Of course, you could move this out to the top-level scope of the file, but I don't like to pollute the global scope like that.



// MyPage.js (Parent Component)

import React from 'react';
import NextBreadcrumbs from "./NextBreadcrumbs";


function MyPageLayout() {

  // Either lookup a nice label for the subpath, or just titleize it
  const getDefaultTextGenerator = React.useCallback((subpath) => {
    return {
      "post": "Posts",
      "settings": "User Settings",
    }[subpath] || titleize(subpath);
  }, [])

  // Assuming `fetchAPI` loads data from the API and this will use the
  // parameter name to determine how to resolve the text. In the example,
  // we fetch the post from the API and return it's `title` property
  const getTextGenerator = React.useCallback((param, query) => {
    return {
      "post_id": () => await fetchAPI(`/posts/${query.post_id}/`).title,
    }[param];
  }, []);

  return () {
    <div>
      {/* ...Whatever else... */}
      <NextBreadcrumbs
        getDefaultTextGenerator={getDefaultTextGenerator}
        getTextGenerator={getTextGenerator}
      />
      {/* ...Whatever else... */}
    </div>
  }

}

This is one of my more in-depth and technical posts, so I hope you enjoyed it. Please comment or reach out regarding any issues to ensure consistency and correctness. Hopefully, this post taught you a few strategies or concepts about Nextjs.

If you liked this or my other posts, please subscribe to my brand new Newsletter for weekly tech updates!

Switching Jobs in Tech Industry: 6 Key Lessons I've Learned

Daniel Starner — Thu, 04 Nov 2021 21:22:40 +0000

Switching jobs in the tech industry can be an interesting adventure, especially if you've had years of experience at your previous position. The dramatic change can spur many emotions, experiences, and situations in your life; both good and bad. Since I know many are either actively going through this process or are considering it I figured I would leave my thoughts.

For context, I recently went through a job transition and started working at Heroku - aka Salesforce - this past September. I won't go into why I left my previous position, but I want to discuss my discoveries once I started to get settled into my new position. I'm only ~51 days into this position, so while there is a lot more to learn, I find my immediate experiences to be most helpful for those considering switching.

Con: Productivity Will Be Tough at First

During my transition phase, I decided to take almost a full month off between jobs. During this time, I went to Disney, played quite a bit of video games from my Steam backlog, and increased the amount of time that I was biking & exercising. It was all great! I felt like I had been revitalized as I was leaving behind some of the craziness of my old job, and I felt almost like a kid again with limited responsibilities.

The downside to this time off was that it was very difficult to get back into the groove of working again, as I struggled to find the motivation the first few weeks to actually want to work. Obviously the paycheck 💰 is the ultimate motivator, but it was still an adjustment to get back to always logging on 9-5 when I was getting used to being able to do whatever I wanted during the days of my unemployment.

I will say that this process was made easier by switching to a fully remote position with a globally-located team. Since everyone on my team lives in different timezones, it made the work-day hours less strict, and I feel more able to get away from the keyboard in the middle of the day to do something for myself, like a bike ride or a nap. Now that I've been around for more than a month, that productivity and motivation is back, but wow, it was rough at the beginning.

Grass is Always Greener on the Other Side

Without getting into specifics, I left my job because I was unhappy with something in their current position and I saw the opportunities of a better existence in a new environment. While I understood that no job was going to be perfect, I didn't realize the number of similarities that different companies shared, both good & bad. Many processes, experiences, and mindsets were similar to how my old job operated.

It was nice to know that I could already be familiar with some things, but there were other places where I could build off that knowledge to learn the unique aspects of my new team. The two positions weren't as different as I had anticipated though.

There was one area that I let myself be blind to during the transition process...

Politics Exist Everywhere

Without going into details, one of the reasons why I decided to find a new position was due to some internal bureaucratic decisions that I disagreed with. I got tired of playing politics for certain things that I thought were self-explanatory, and organization changes were occurring that went against some of the foundational ideals that my department had spent years building up.

I had told myself, "Hey, these B.S politics are getting old, let's go somewhere else!" and I quickly realized that every job has shitty politics and bureaucratic decisions to deal with. It's nothing against these companies, but at some point, business decisions have to be made and those are going to clash with engineering desires. Now, with that said, I believe that there is a line that can be crossed where one shouldn't keep putting up with politics, but to have the expectations of never dealing with politics?...that is just unreasonably in the modern workplace.

With this conclusion, I realized that I could change the angle that I was viewing some of these problems through. While I think the switch was still ultimately needed for me, I believe I've grown and have learned how to handle some of these situations better.

One Man's Trash...

I find that the old adage of "One man's trash is another man's treasure" is pretty true concerning jobs during the Great Resignation. I saw pretty high levels of attrition at my old job - heck, I was one of those 😉 - and my new manager was pretty open that the team I was joining just faced some higher-than-average attrition levels.

Does that raise red flags? 🚩🚩🚩🚩🚩🚩🚩 Hopefully not if you've done your due diligence before joining the company. I knew that this was occurring, and I knew the situation was changing; not for better or worse, it was just changing. Changes meant new challenges and potential technology refreshers; it meant that I had the potential to make a bigger difference, and I was drawn to that opportunity.

One thing that really excites me about starting a new job is that I can bring fresh perspectives to projects that may need it. If the same people are working on the same project for an extended period of time, it starts to create a silo of ideas and knowledge. By introducing new people to these projects - aka me onboarding - then these projects can get some new life. This allows them to be viewed in a way that allows us to make any needed changes that might have otherwise been overlooked.

Be Open to New Things

"Of course I'm open to new things, I just quit my job for heaven's sake!" I tell myself as I write this. But really though, I switched jobs and immediately was thinking about how to jump on a project that is most similar to what I'd been doing previously. I worked with a lot of Kubernetes cluster management at my old job, and I really wanted to just jump right into more Kubernetes projects.

Wanting to work with a specific technology is okay, but I wasn't really giving myself a full chance to grow more. With a new team & company came new opportunities, and I quickly realized that I was limiting myself by having such a small view of what I wanted to work on without knowing the full scope of projects and available capabilities. Once I saw myself doing this, I started giving other projects more of my time, and I found that I was learning a considerable amount in areas that I had limited familiarity in. This led to some of the confidence checks that I list below, but for the most part, its already been a pretty neat journey to connect the dots between my previous experience and the stuff that's totally new to me. By tackling these new problems, I am becoming a more well-rounded engineer, and I can feel my confidence raising by a considerable amount in these areas.

Be Confident without Faking It

So this section depends on what level you were at your previous company vs what you are joining as, but I feel like there's something for people of any level.

For my situation, I was a newly minted Senior Engineer when leaving, and so I applied & received a Senior Engineer role at my new company. Do I deserve to be a senior right now? Well, my Imposter Syndrome says that I'm not good enough, but I've led projects, maintained critical systems, and built large complicated systems, so here I am as a senior. 🤷🏼‍♂️ I've solved enough problems and pulled out enough of my hairs to justify being a Senior - I'm looking at you Jenkins/Groovy for most of those hairs pulled!!!

Being a senior brings with it a certain level of expectations and responsibility, which I was familiar with in my old job. The issue was that I was attempting to maintain that same level of expectations in a job where I barely knew what was going on. Most companies and teams expect onboarding new engineers to take at least a few months - based on the complexity of the system(s) - and my new team was no different. For some reason though, I was trying to shortcut this timeline and prove my skills as an engineer to those around me as quickly as possible, which led to some failed personal expectations and confidence issues that made me question my own abilities as an engineer.

// Detect dark theme var iframe = document.getElementById('tweet-1455300847328534531-311'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1455300847328534531&theme=dark" }

Note: This isn't the blog post in question, but maybe I'll still post that other one implied in the Tweet 🤔

Don't Stress About Proving Yourself

If you are like me who puts a lot of pressure on themself to succeed, then switching jobs will probably lead to some stress & Imposter Syndrome from facing problems against an unfamiliar position. It's okay to struggle the first few months in a new position. Like I mentioned, teams expect around 3-6 months to onboard engineers, so don't expect to start dropping big Pull Requests by the end of your second week.

I wish I told myself this earlier though. I've been in this position for less than 60 days and I've already almost burned myself out 😅 Why? Because I set an incredibly high bar for myself where I tried to match the work output that I had at my old position. The issue there is that I don't have the 3 years of institutional knowledge at my new job that I gained at my old job. During the beginning of my onboarding, I was constantly telling myself & my manager that I needed to get some big contributions out as soon as possible to prove I was worth hiring.

It's almost impossible to hit the ground running when starting a new job though. For example, its difficult when you have a system and multiple environments as complicated as Heroku's tech. Note: This is nothing against Heroku, its just **impossible* for anyone to learn a large company's tech stack within 2 months*.

As time progresses, I will get more comfortable with the systems and my productivity will rise. It's going to be bad for my mental health and productivity to assume that I will be able to solve all of my new problems with the same ease that I did at my old job. My manager & team are not putting that level of pressure on me, so why am I putting it on myself? This is still something that I struggle with occassionally, but being away of the problem is the first step to solving it, and I just need to take a deep breath and let myself relax a bit.

Pro: New Colleagues

Switching has allowed me to get a brand new group of work peers! I worked with some really great engineers at my old job, and my new position is no different. Having the opportunity to work, learn, and grow with some of the best engineers in the world is something that I do not take for granted. While I'm not surprised by the fact that I am growing relationships to my new coworkers, I am surprised at how quickly it's happening. I already feel like I have the support of my managers, architects, and colleagues, and I am gaining their trust more and more as time goes on.

I really want to keep in touch with many of my old peers, and I'm sure I will to some extent, but I'm excited for the challenges that I'll be able to overcome with my new team. I just need to meet all of them first, which is okay, because I've been trying really hard to reach out and schedule 1:1s with everyone.

Initiate 1:1s with Everyone Around You

An easy way that I started to build relationships with my team was by reaching out and setting up 1:1 introduction meetings with many of the people I work with day-to-day. This included those on my immediate team, sister teams, and my management chain.

I tried to schedule meetings with the following approach, and I think it worked/is working pretty well!

First 2 weeks: Meet with my manager and immediate teammates
First 4 weeks: Meet with some members you are interacting with that are on sister teams
First 6 weeks: Meet with any senior/principal engineers in your work tree and architects if those exist at your company

Heroku also uses Donut to connect different people in the organization, and I've been trying to actually schedule meetings with the other individuals the app pairs me with. It's made it somewhat easier to introduce myself across the organization, and to meet people in a way that is normally difficult for a remote-only workforce.

Bonus: Write Down Notes for EVERYTHING

So I thought of this extra note about my onboarding experience, because I'm usually the one writing documentation or explaining a technical solution to others, and it's been a fun experience to turn the table on that notion as I get to learn everything my team interacts with.

Sometimes i feel like I am asking too many questions, but I recognize that is just a part of the onboarding process, and I appreciate my colleagues' willingness to always help me get the answers I need to understand the systems a little bit better. To keep this process more enjoyable for them though, it's important that I note & remember as much of the information they give me as possible, so that I don't need to ask again. This note-taking process also helps, because it provides the opportunity to convert these notes into real documentation that can be used by others who find themselves in my same position...then maybe they'll be asking me questions. 👀

As an extension, I'm glad that I found Obsidian right as I was starting my new position, because my gosh I would be lost without a good note-taking app. Everything from onboarding requirements, to technical notes, to people's names/positions...you name it, and I have it in some notes from my onboarding. This has helped me stay on top of things while I start to climb the mountain of knowledge that my team works with, and I find that writing stuff down really helps me remember important details.

I am excited to continuing growing in my new position, and I will probably have more posts to write about in the future regarding the cool work I get to do, but for now, I'm excited to just learn. The amount of information I'm getting to learn in my new position is truly incredible; it's like trying to drink from a fire hydrant sometimes. We'll see what I do and where the future takes me, but this transition period has been quite an experience, and I hope that some of my thoughts help you, wherever you may be in your own journey.

Introspecting Python Parameter Values via Argument Binding

Daniel Starner — Sun, 17 Oct 2021 01:23:07 +0000

Sometimes it is important to map function arguments with their parameter values. Python offers basic keyword arguments - which will be discussed below - but if arguments are passed either positionally, or are handled by "catch-alls", such as *args or **kwargs, then it quickly can become more difficult to deterministically map parameter names to their associated values in a function call.

Some situations where this may be mapping may be desired include:

Observability tools and call stack tracing - Seeing what functions were called with what methods helps to determine how efficiently a program is running, or can help debug it when things go wrong.
Determining function/task ownership for audit purposes - If you system is a product used by organizations and users, it is important to have an easy, low-overhead way to determine what resources are being affected by a function call and who requested the change.
Argument modification - Sometimes a function's arguments need to be validated or modified before being used, and creating a decorator that can handle any arbituary parameter definitions may be useful.

In this post, we will dive into some more magical Python libraries and utilities that allow us to wrap and perform certain operations against arbituary function parameters. I will start with some Python basics - feel free to skip those sections as needed - before diving into some of the Signature magic and it's real world use cases.

Functions Review

Python, like most other programming languages, has the ability to define functions. These functions usually define input parameters and they output return values. Argument values can then be provided to the function, which will perform some work and return a value. Note that I will try to use Python annotations where possible to make parameter types more clear.

def add(num_1: int, num_2: int) -> int:
    """Adds and returns the two arguments
    """
    return num_1 + num_2

add(5, 7)  # returns 12

Quick note: Parameters are the definitions of what variables a function defines, and arguments are the actual runtime values that satisfy those parameters. In the example above, num_1 and num_2 are the parameter definitions, and the values of 5 and 7 are the arguments passed to the function.

Functions in Python can define arguments in two common ways: as positional arguments and keyword arguments.

Positional Arguments

Positional arguments are defined at the beginning of the parameter list, they are matched with their parameters based on the order they are provided, and they have no default values. A positional argument must be provided to the function when called.

In the example above, since the function defines num_1 and then num_2, it means that the first argument passed (5) will be saved to num_1 and the second argument passed (7) will be saved to num_2.

Keyword Arguments

Keyword arguments are defined after positional arguments, and they can provide default values in the parameter definition, meaning that a matching argument does not need to be provided when the function is called. These arguments are provided using the parameter identifier, which is the variable name given to the parameter. If they are given positionally, without the identifier, then they act in the same was a positional arguments.

We could call add using keyword arguments:

add(num_1=10, num_2=8)  # returns 18

Notice how we have <identifier>=<value>. We are now using keyword arguments to set num_1 and num_2.

Default Keyword Arguments

add doesn't require keyword arguments by default, so lets consider the following function:

def exponential(num, power=2):
    return num ** power

The power parameter above is defined as a keyword argument by default, where if its not provided, it will be given a default argument value of 2.

exponential(2)           # returns 4
exponential(2, 3)        # returns 8
exponential(2, power=3)  # returns 8 as well

Problem Statement

Knowing how arguments and parameters work in Python, what if we want to write a program that prints out what arguments are given to a function, lining up positional and keyword arguments to a dictionary mapping. We will create a decorator that accomplishes this.

We are going to write the log_function_call decorator function that will print out the arguments and return value of any function wrapped by the decorator.

@log_function_call
def pow(num, power=2):
    return num ** power

pow(5)
# -> "pow was called with { num: 5, power: 2 }"
# -> 25

pow(5, 3)
# -> "pow was called with { num: 5, power: 3 }"
# -> 125

pow(2, power=4)
# -> "pow was called with { num: 2, power: 4 }"
# -> 16

To create this, we will need to solve the problem of determining what arguments given line up with the parameter definition in the function, so that we can print it out. In a more general sense:

How can we intercept and determine what values are passed for a Python callable?

In this case, a callable is anything that can take arguments and return a value, such as a function, method, or even a lambda function.

Python's Signature Class

Python has a built in inspect module that provides many different utility functions and classes which allow the caller to get information and metadata about runtime variables during a program's execution. We can leverage this module and it's Signature class to introspect a function's arguments to create a mapping of parameter names back to their values, even if they are passed as position arguments!

Without the Signature class, only keyword arguments can be easily converted into a dict map, which limits the amount of introspection that can be performed. By using the Signature class, a generic decorator, such as log_function_call, can convert any arguments given to a function into a map of their parameter names and values. This is powerful as it allows us to create a more powerful decorator that can perform extra actions using a function's arguments, whether passed positionally or as keywords.

Quick Look at Decorators

Before diving into how the Signature class can help us with the log_function_call decorator, lets start setting up the decorator itself so that we can understand that code before going further. While I assume some basic knowledge of Python decorators, I will briefly explain the code in what the decorator block would look like.

def log_function_call(func)

   def wrapper(*args, **kwargs):
       """
       Wrap the original function call to print the arguments before
       calling the intended function
       """
       print("TODO: Print the arguments here!")
       func(*args, **kwargs)

    return wrapper

By wrapping a function with this log_function_call above, we will emit a print line before the function actually gets called. The only issue is that we are printing an unhelpful line instead of our actual argument values.

Signatures and Binding

So we have a decorator that is intercepting the function call with access to the arbitrary positional argument - as a list represented as *args - and keyword arguments - represented as the **kwargs map. By generating a signature of the arbitrarily wrapped function - func in the example - we can merge these two generic argument objects into a single, declarative map of

<parameter name>: <arg value>

which can then be acted upon easily to perform different operations, such as printing out the function call and arguments for auditing sake!

In the following code, we are continuing off the previous decorator block with the addition of our introspection code; we are generating a signature of the wrapped function and then binding the given arguments to the functions parameters, which returns a dictionary of parameter names to their passed values.

from inspect import signature


def log_function_call(func)

    def wrapper(*args, **kwargs):
        """
        Wrap the original function call to print the arguments before
        calling the intended function
        """
        func_sig = signature(func)

        # Create the argument binding so we can determine what
        # parameters are given what values
        argument_binding = sig.bind(*args, **kwargs)
        argument_map = argument_binding.arguments

        # Perform the print so that it shows the function name
        # and arguments as a dictionary
        print(f"{func.__name__} was called with {argument_map}")
        func(*args, **kwargs)

    return wrapper

Real World Example

So when can this actually be used? Well, for our project, we have a Django web application project that implements Celery Asynchronous Tasks. We can schedule these tasks to be run by a background worker, but we require an audit log to determine who started a task and what organization the task is tied back to.

For instance, if we have a task such as resolve_membership that ensures the right members have the proper permissions, it may take in a organization_id parameter of the organization to resolve membership for.

from celery import task


@task(bind=True)  # <-- Fanciness that just denotes the function as an async task
def resolve_membership(task: TaskResult, organization_id: int):
    org = Originization.objects.get(id=organization_id)
    do_something_with_the_org(org)  # arbituary code being run

While this task is created, scheduled, started, and competed, it's current state and result (SUCCESS, FAILED, QUEUED, ...) will be stored in the database by Celery automatically. This gives us a queriable audit log of all resolve_membership tasks that have been run, but it doesn't easily actually allow us to see which tasks correspond to which organizations. As a part of task scheduling, we wanted to create an audit log of these tasks and draw a relationship to what organizations they were affecting, and what - if any - user scheduled the tasks.

We could have performed this audit log creation explicitly in every task, like the following code example, but it would have gotten more difficult to manage as more tasks were created, or if any changes were made to the custom TaskResult class. This would quickly become too cumbersome and would not result in a DRY program structure.

from celery import task


@task(bind=True)  # <-- Fanciness that just denotes the function as an async task
def resolve_membership(task: TaskResult, organization_id: int):
    org = Originization.objects.get(id=organization_id)

    # Assign the organization to the task for audit logging purposes
    #   NOTE: this would need to be done on EVERY task
    task.organization = org

    # Perform the actual task needed
    do_something_with_the_org(org)  # arbituary code being run

Using a wrapping decorator like the log_function_call above, we were able to create a single assign_ownership decorator that would wrap each task execution. This decorator would inspect the task arguments and provided some static arguments to the decorator itself, it would be able to create the same ownership relationship without needing to copy anything more than just the decorator name itself. This made it much easier to create the task audit log without having to worry about how the ownership was actually being created. It reduced all of our individual implementations across tasks to a single function block that was much easier to manage.

from celery import task

@task(bind=True)
@audit_ownership(org_param='organization_id')
def resolve_membership(task: TaskResult, organization_id: int):
    org = Originization.objects.get(id=organization_id)

    # Perform the actual task needed
    do_something_with_the_org(org)  # arbituary code being run

The following code block is a paraphrased version of the decorator. Notice how we are using the Signature class to load the value of a given parameter no matter how it was passed as a function; whether as a positional or keyword argument. Note that the following code is a bit advanced and does require some knowledge of how to define decorators in Python.

from inspect import signature


def audit_ownership(org_param: str, task_param: str ='task'):
    """
    Wrapping a celery task, this will attempt to create an audit log
    tying the task execution back to an organization using passed arguments

    : org_param:  str  : function parameter that denotes the organization ID
    : task_param: str  : function parameter that denotes the task instance
    """

    def decorator(func):
        """
        This is the actual task decorator, but nested to allow for parameters
        to be passed on the decorator definition
        """

        def wrapper(*args, **kwargs):
            """
            The wrapper replaces the actual function call and performs the
            needed extra auditing work before calling the original function
            """

            # create a function signature to introspect the call
            sig = signature(func)     
            # Create the argument binding so we can determine what
            # parameters are given what values
            argument_binding = sig.bind(*args, **kwargs)
            argument_map = argument_binding.arguments

            # Using the argument binding and decorator arguments, we can
            # fetch the task and organization, no matter how the function
            # was invoked. The power of argument introspection!!
            task = argument_map[task_param]
            organization_id = argument_map[org_param]
            organization = Organization.objects.get(id=organization.id)

            # The actual logic is abstracted out to be more readable
            # Assume this function creates the relationship between the
            # task result instance and the organization instance
            assign_ownership_to_task(task, organization)

            func(*args, **kwargs)            # this calls the original function
                                             # as it was original intended  
        return wrapper

As an extra bonus, we made the org_param decorator parameter also be a callable, instead of being a str, so that we could dynamically fetch the associated value. This was useful in situations where organization-owned resources were passed to the task instead of the organization itself, so we could have the callable load the child object and return a reference to its parent's organization.

Python decorators that modify and perform operations using the provided arguments can rely on function signatures and argument bindings to determine parameter values as a part of their operations. This allows programs to utilize generalized decorators that can be reused across an application to perform operations such as debugging or audit-logging. Have some more examples of where this could be useful? Post them below!

Maintaining Quality Documentation

Daniel Starner — Thu, 09 Sep 2021 13:25:14 +0000

Communication is difficult. As remote work becomes more of a thing, as teams work across six+ timezones, and as companies are looking for rapid growth, it is paramount that technical knowledge is able to be communicated correctly and effectively.

Over my past few professional experiences, it became apparent that I had started to spend more of my time communicating ideas and interfaces with others, and less time actually developing them. Now, obviously the design phase before building something is critical, but I was more concerned about the amount of time I spent discussing a feature after finishing it. I found that if I spent some time building a new feature for our product, I would spend at least two or three times the amount of time answering Slack messages for users who wanted to use it or who had questions.

I realized that while we had good technical engineering practices, we were lacking the communication channels & skills to allow our clients to effectively use the product. This resulted in a fair amount of precious engineering time lost to our on-call team member basically acting as tech support on Slack, as they'd have to answer very similar questions multiple times throughout the day.

The Curse of Success

Our product was still young, but it was quickly growing in popularity at the company. It was a cluster configuration and deployment system that managed many of our internal services. I was one of the lead engineers responsible for developing this ecosystem, and we had built out quite a few systems and solutions to solve all different categories of technical problems.

We were able to implement these problems very well, but we had grow so fast that we started to realize some problems with the way we were communicating and sharing our product with the rest of the company.

As more users found the tooling and started using it, we had more users who ranged from "Oh this is neat", to our advanced users & internal team members who knew the full "in's and out's" of the tools. Somehow, we had to make sure every end user and effected client could effectively use our tooling to solve their deployment problems.

This growth problem was easily manageable when we had just a few teams using the product, as many of those users had been with us since the start of the product and could use it moderately well. We had a page or two of loosely organized Jekyll documents that briefly explained how to get different environments up and running, and the users were expected to sort-of just figure out the rest.

Our documentation at the time was like the Owl-drawing tutorial below; we will walk you through the left-side picture, and then just throw you to the fire and hope you figure out all the details as you go to production.

Tribal Knowledge and the Silo of Doom

Tribal knowledge is any unwritten information that is not commonly known by others within a company.

We quickly realized that our team and the original user base had quite a bit of tribal knowledge in the system. As our product grew in popularity and was used by new users, they did not have this knowledge that seemed "trivial" to us, but in reality was confusing for these first-time users. We had no way to effectively communicate the silo of knowledge that we had developed over the years of building & using the product. It's very difficult for people to use something if they don't know how to use it. They are even less likely to use it if they see others describe it as "easy" without seeing the path to attaining that knowledge. When people notice this gap of knowledge between it's users with no discernible way to resolve it, the reputation of the product is at stake.

Our users wanted to learn our product, and we wanted them to love the product, but we had a documentation problem.

Cue the Documentation Attempts

Our team set off to close this documentation gap between our users and tooling. We had three main areas of tooling & associated documentation that needed addressed:

A web user interface that provided a nice overview and data visualization of our system
A RESTful API that provided all of the deployment data and configurations for the web UI and any integrated services
A CLI for command-line-based operations which allowed for local development and cluster configuration management
Our overall concepts and tutorials so that users could learn the system from a high level across the aforementioned tools

It was important for us to solve our documentation problem for each of the services above, because we know that good project documentation allowed the following situations:

Improved onboarding of new users and new team members
Reduction in support questions by users
Ability to quickly link users to documentation sections for answers and concepts
Increased reputation and usage due to (hopefully) being more user-friendly

Documenting the Web Interface

Web interfaces are hard, because they should be designed intuitively and cleanly so that users familiar with the interface can interact with it as efficient as possible. This presents a challenge for newer users who are unfamiliar with the system, as the layouts and functions of components will be totally new to them.

To overcome these documentation issues in our web interface, we had both proactive documentation through means of initial popups that would call out certain elements on the first visit to a page type, and passive documentation that would open a sidebar containing page-specific documentation when an icon button was pressed. We felt like this approach was a good trade-off of introducing new users to elements without being too intrusive.

The highlighted element pop-ups used Reactour to display their information. To ensure that we covered future changes to the steps, we would hash the list of generated steps and save them to the client's localStorage in the form of [page URL]: <steps hash> where the [page URL] was of generic form, such as /settings or /users/[user_id] so that we only showed the steps once per dynamically-rendered page type. These pop-ups provided very basic introductory information about the elements that they highlighted, such as what functions they performed.

elrumordelaluz / reactour

Tourist Guide into your React Components

For our passive, yet detailed documentation, we allowed each page to pass a markdown page source to our core layout component that would lazy load and render the markdown into a nice right-side drawer that could be toggled open and closed. This sidebar would be used for presenting more details about the overall page usage and components that existed.

Documenting the RESTful API

Our RESTful API was actually the easiest to document, because we were able to rely on third-party libraries from the start. It was a Python Django and Django REST Framework project that leveraged the drf-yasg OpenAPI generator library to create OpenAPI and Swagger compatible documentation.

axnsan12 / drf-yasg

Automated generation of real Swagger/OpenAPI 2.0 schemas from Django REST Framework code.

This library allowed us to write the API documentation in-place throughout the route definitions, and drf-yasg would render documentation in either the form of an OpenAPI schema or as an interactive Redoc/Swagger UI web page. This meant that we were generating documentation from our code and there was no extra developer steps to change documentation if a feature or route changed.

Documenting the CLI

Our command line interface was a Golang binary executable that was based off of the cobra and viper libraries which made creating advanced CLIs very easy.

spf13 / cobra

A Commander for modern Go CLI interactions

Amongst the many neat features of cobra, we were able to extend the Markdown documentation generation feature which would create rich user documentation for each of the available commands. We would generate this documentation at release time and either deploy the static files as a GitHub Pages site, or integrate them with our generic documentation below. We made sure to always include older release documentation for users who were not on the latest CLI version.

Documenting the System

With the individual components laid out, it was finally time to document the entire system so that it made sense to users. We actually jumped through three or four documentation generators over the three years that I worked on the project.

The loose timeline of our documentation services was:

Jekyll → Hugo → Docsify → Docusaurus

Jekyll

As mentioned earlier, our documentation started in Jekyll, and it worked decently well for us. The two main drawbacks for us was that

It slowly grew too difficult to manage large documentation as links & page references needed to be manually updated, and it became more difficult to manage as the number of pages grew
Our team did not use Ruby, so the development process of putting docs together put users on an unfamiliar path.

We also found that some of the documentation templates seemed to be dated, and we were just looking for something that looked more refreshed and clean, but that's more of an subjective reason to switch than a technical one.

Hugo

We gave Hugo a shot next, and it provided some more validation and speed in terms of development time. Since the associated program is just a binary, it could be very easily installed on our machines and in workflows without much fuss.

Hugo provided more flexibility and validation across our documentation, and we quickly grew our documentation and made it look much more organized using the Hugo Learn theme.

Hugo was great, but it required a Build step to be run between pages in a GitHub repository could show up in a GitHub pages site. We built workflows to handle this on merges to main, but it wasn't perfect and would occasionally fail. This generation step had us looking at client-side generators again, and we ran into...

Docsify

Docsify is a client-side only documentation generator. As a user requests a documentation page, it would fetch the associated .md file and render it all client-side. That meant that our documentation deployment process was simple again, as we could just push changes to the main branch and BOOM they would show up within seconds.

The other draw of Docsify was that there were some really slick minimalist themes that we found and really enjoyed. We could pair these with the long list of extensions and provide users with clean, yet extensive documentation.

The drawback with Docsify was that as our documentation continued to grow, it started to slow down the rendering process as the initial load would require fetching a lot of information. On top of this, the searching functionality became more unusable as it wasn't a great interface, and it had to search all files in a flat format to generate the results. My final gripe with Docsify was one of the reasons that I was initially drawn to it; anything more than generic markdown requires plugins. This introduced a bunch of documentation dependencies on libraries that didn't seem fully legit and managed, and we found ourselves having to write custom plugins a lot to make things render as desired.

Docusaurus

After jumping from documentation generator to generator, we finally settled on Docusaurus. It introduced a build step again similar to Hugo, but we found that the trade off was worth it, as Docusaurus brought a bunch of really nice features with it. Since everything was just React under the hood that could be customized, we naturally gravitated to it as we already had React experience.

We found that it was much easier to customize and extend Docusaurus as compared to our other generators. This meant that we could provide the full & rich experience that we wanted. Our users also responded the best to this iteration of our documentation, but that may partially be accredited to us spending more time to organize the content as well as the overall layout.

My Documentation Tips

The following tips and suggestions are just a few of the ways that our team overcame our documentation debt challenges, and how we ensured that our documentation was always in an acceptable state, no matter the level of user who was reading them.

Generating Documentation from Code

As previously mentioned, our API and CLI documentation was generated directly from the underlying code. I highly recommend this strategy of Documentation-from-Code; it makes the process of writing great documentation that much easier. Having one interface for developers to add and document features makes it more likely that they will actually write documentation.

Most languages and application types have some set of libraries to assist in generating documentation from code, and I highly recommend implementing them in your user-facing projects.

Writing Less More Often

It's difficult to climb a mountain in a day, but it becomes easier if you take one small step periodically. Documentation works the same way; writing all of your team's documentation at once will most likely cause stress and annoyance. It's strongly encourages to write documentation in small increments, hopefully at the time that the associated feature is written.

For older features that need documentation, don't fret; just try to write a few paragraphs, or even sentence, whenever you have some free time. As long as you are slowly crunching way at the documentation debt, you are improving the situation.

Finally, I found that it was easy to add/improve documentation for a certain feature if a user had just asked a question about the topic. This showed either a gap in our documentation, or a discrepancy that confused users. By tackling these issues one at a time, it was easier to ensure good documentation.

Organizing Documentation by User Story

As our product matured, our documentation evolved from a list of individual available features to tutorials and guides based around common user workflows. We had a section of tutorials for our common use cases that included users just getting started with the system, all the way to some of our most advanced use cases.

Being a deployment system, we broke up our documentation into cluster timeline user stories:

Day 0: Gathering and provisioning infrastructure
Day 1: Implementing cluster tools such as monitoring and alerting
Day 2: Deploying application and configuring DNS amongst other things

This layout provided users with a pretty linear approach to reading our documentation. They could either get the brief steps through a tutorial, or they could follow our guides in order to use the system. Our support questions have reduced slightly since we switched to this more linear documentation organization.

Devoting Time for Documentation Debt

It's not too unheard of to hear about engineering teams having hackathons during the workday to allow contributors to work on neat projects and squash technical debt. I would say teams need to go one step farther and have Documentation Hackathons, or at least give individuals the ability to fully devote time to both their client-facing and internal documentation.

This allows some time to "refresh" the content and ensure its fully up to date.

Good documentation practices are key to ensuring happy users and keeping technical support questions to a minimum. Hopefully the above experiences, technologies, and tips help you and your team present your products as positively and completely as possible!

Updating My GitHub Profile README

Daniel Starner — Wed, 05 Aug 2020 05:06:54 +0000

A few weeks ago, GitHub released full support for adding READMEs to profile pages. This allows users to spice up their pages with greetings, contact & professional information, or to create a fun, random game of chess.

// Detect dark theme var iframe = document.getElementById('tweet-1286724102099816448-215'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1286724102099816448&theme=dark" }

I've been meaning to create one for a few weeks, and apparently midnight on a Tuesday was the right time for me to actually go through with it. All it takes is to create a new repository named after your username with a README.md file in it! My profile README includes the following:

A nice introduction to greet people
Some quick facts about myself to get people up to speed
A disclaimer that my public GitHub code may not be up to par with my best skills 😅
My current projects, volunteering efforts, and hobbies
My latest Dev blog posts (credit to @gautamkrishnar for the workflow)
My GitHub stats across repositories

You can visit my profile, or just check out the screenshot below. What kind of stuff have you seen, or what does your profile README look like?!

What Tool Can You Never Remember or Get Good At?

Daniel Starner — Thu, 05 Mar 2020 01:16:46 +0000

What program, application, or shortcut have you never been able to get behind, even though most people tout up as the best thing for developing? It could be either because you don't care to learn/use it, or because you always forget how to use it.

For me, its tmux. I realized it today when I kept complaining about my ssh session crashing and having to start again. My teammate reminded me that tmux has persistent sessions, but after 3 or 4 times of trying to use it more often, I just find it frustrating and never really use it. Maybe one day I'll actually get good at using it, but for now it just seems like a lot of trouble and time that will slow me down in the short term.

Some other honorable mentions:

vim / emacs: I know the basics of vim, but holy heck am I slow at making changes
curl: I basically need to relearn curl every time I want to make a simple request from the command line

The Myth of Sisyphus, Failure, & the Meaning of Imperfect Code

Daniel Starner — Wed, 26 Feb 2020 02:12:07 +0000

For years as a student and into my first few years as a junior engineer, I felt dirty. Something felt wrong about the code I wrote. No matter how hard I tried, things just didn't always line up and it bothered me. "What is the point of writing code if its not the best it can be?" I would ask myself. If my editor was a pencil and paper, I would have gone through all of the world's supplies of erasers before ever coming to a solution, because I felt like the code I was writing wasn't good enough to give to a client or to deserve a job. This made certain parts of my early career miserable, slowed down projects, led to burnout, and even made me reconsider my life choices a few times.

As I've grown and learned from the experienced individuals around me, I've come to accept my imperfect code and imperfect solutions. While coming to this acceptance, I was able to draw some notable comparisons to one of my favorite ancient Greek myths, the tale of Sisyphus the king. He was tasked with rolling a boulder up a hill for all of eternity only to have it fall back to the bottom, all because of a con he attempted where he tried to cheat death. Sounds pretty grim, right? But there's a lot to learn from this myth and a lot to be optimistic about when digging deeper into it. Even thousands of years later, we can relate this myth to the daily grind of software engineering and programming.

A Note Before Going Further

“There are three things I have learned never to discuss with people... Religion, Politics, and The Great Pumpkin.”

Charles M. Schulz, Creator of Peanuts

I'm going to mention a piece of writing throughout this post titled The Myth of Sisyphus by Albert Camus that talks about existentialism and the absurdity of life. I cannot stand on a soap box and argue whether or not Camus's thinking on life, death, and the meaning of it all is correct, I simply find the story interesting and drew some conclusions to my own thinking about the Software Development Life Cycle and what it means to be a Software Engineer. It is however an extremely interesting book no matter your background, so I recommend giving it a read if you are looking for some crazy deep philosophy and just a few minor cases of existentialist dread.

Two Common Scenarios of Mine

The two following situations seem to occur a lot as a junior engineer, and I know many colleagues and former classmates who let these situations get to them, chasing them away from the software industry.

Changing & Misunderstood Requirements

Have you ever had those days of programming when you seem to be on a roll, and then suddenly you realize any one of the following: 1) you missed an important edge case, 2) misunderstood either the problem or the solution, or 3) the requirements were changed out from under you? After working long enough in the software world, one, if not all three of these situations will happen to you, and its just upsetting, especially if you have already have been deep into the project or a particular implementation. This causes you to reevaluate, start over, and reset your entire thinking on the problem, potentially burning away hours or even days of time from your previous attempt.

Solution Stinks of Code Smell

"Code smells are a set of common signs which indicate that your code is not good enough and it needs refactoring to finally have a clean code."

Mohamed Aladdin post on Writing Clean Code

Quite frequently I will design a solution that seems perfectly reasonable and easy to explain in my head, only to find that it is difficult to implement in real life. I'll tell myself something along the lines of "Oh yea, I'll just create class X and parent interface Y and the code will be awesome!"

...then you go to write this perfect implementation, annnnnddddd just nothing seems to come out as well as it looked in your head? Either interface Y just doesn't seem to be working, or maybe the dependency injection you thought would be smart in class X is backfiring in your face.

Code is the Boulder

"[A]ccording to the Greek myth, [Sisyphus] was punished for all eternity to roll a rock up a mountain only to have it roll back down to the bottom when he reaches the top."

SparkNotes on The Myth of Sisyphus

Don't tell my high school literature teachers that I quoted SparkNotes, but as previously mentioned, Sisyphus's punishment was to role a boulder up a hill, only to watch it roll back down as he approached the top. After all the time and effort he put in, he would watch himself fail and have to start back from the beginning again.

The aforementioned mental blocks and difficulties are the problems that I face in my personal and professional life; they are my boulder(s).

Software development is a lot like the myth of Sisyphus; it is never-ending, is always pushing forwards, and has the potential to hand you big failures when you are least expecting it.

I think the Dev user jacksonelfers says it pretty well in his comment below.

Jackson Elfers • Mar 6 '19

Programming is a lot like the Greek mythology of Sisyphus. Never ending, never complete, sometimes things go up in flames you were completely confident about. Then there are moments that remind you the perks of being able to talk to computers.

Our responsibilities with programming are never-ending, and sometimes burn us. Even if we "complete" a project, there may be bug fixes, patches, new features, or at least another project behind it waiting to fill more time and space in our lives. In this regard, we as programmers are always pushing our boulders up our own hills. There's always tech debt, the next project, the next requirement, or a bug fix that needs to be done. These are our responsibilities and our duties in the software and programming industry.

Iterative Progression, Not Immediate Perfection

Since we know there is always work to be done, it becomes impossible to write perfect code, because perfect code would mean there is nothing left to do. That lifts a massive burden off of our backs to feel like we have to be perfect. Since we know that we will always be rolling our metaphorical boulders up the hill of software development, that opens us up to allow for failures, learning, and so much more.

As a junior engineer, I've messed up projects, missed deadlines, and have seen my "boulder" roll back down the hill as represented by my failures. Previously, I would let myself be defined by these failures, and I beat myself down every time something went wrong. What I failed to notice is that everyone around me - including the hot shot senior engineers - are pushing their own boulders, occasionally failing and watching them fall down the hill.

This is what makes software engineering fun for me now; I recognize that it is a human process. Code is imperfect because we are imperfect. Failing is just a part of the game. Everyone has failed before, and everyone will continue to fail. What makes this struggle all worth it is that every time we fail, we can hope to learn from these mistakes and make it just a little bit further up the hill before failing again.

Writing imperfect code is okay as long as it works, because all code is imperfect. If it works, then that is better than what was there before. There will always be time to redesign and refactor. I love the quote that my manager(s) use occasionally when I fall into the trap of attempting to write perfect code; "Chances are, what you expected to last forever will last a year, and the hack you put in will stay in production 'til after we die." We can't foresee the problems we'll encounter in the future, so we need to just take small steps forward up the mountain to get closer to the top.

This means that instead of reaching for immediate perfection in my projects, I have started to force myself to build iteratively and progressively. Whereas I could have spent six months working on a project in my basement only to realize it sucks, I can spend one month writing a crappy project, release it, and spend the next five months getting valuable feedback and improving upon it slowly and iteratively. That is software development.

We Must Imagine the Programmer Happy

One of Camus's strongest points is that we must imagine Sisyphus is happy pushing the rock up the hill, because it is absurd to think otherwise. Since he knows that is all he can do, he gets enjoyment out of it, because there is no better alternative. Instead of seeing the rock as his punishment, he begins to see the challenge as his life purpose and the boulder as the vessel to accomplish that goal.

As a software engineer, I take great pride in the code that I write. Even though I know most of it will be refactored, rewritten, or won't even be needed in a few years when the next thing comes around, it fills me with a happiness and purpose to know that for the meantime, my code is out there doing some really cool things. Past that, it doesn't matter for me. Even if I commit only a few lines of code to a project, that's better than nothing, and makes me a better developer than before.

The challenges that present themselves in software development are not unlike Sisyphus pushing the rock up the hill. There are challenges, mistakes, and failures that can be represented by the boulder rolling down the hill. While this is true though, I always come back to programming. I always find joy in the challenges and the struggle. Through the grunt work, the nitty-gritty details, and the imperfect solutions, I find meaning and solace that I am making a difference in the world.

Time to Show Off the Robot

Daniel Starner — Sat, 27 Apr 2019 02:16:19 +0000

Now, I usually don't do something like this, but I just spent most waking hours of my life the past 4 months doing nothing but robotics, so I thought I'd show off and be proud of my students...This also explains why I haven't posted since the end of December 😮 This post comes also almost word-for-word from when I wrote it on our team website at https://www.ghouse354.com/2019-robot.

Back in December, I was looking for a new FIRST Robotics team to work with. If you are not familiar with FIRST, I highly recommend looking into it more and working with a team near you. The gist of it though is that high school teams spend six weeks building a 120 lbs robot that best accomplishes the game's goals. This year's game, Destination: Deep Space, was all about moving Hatch Panels (poly-carbonate discs) and Cargo (rubber balls) around the field and placing them in their respective goal positions. You can watch the game video below.

I ended up working with the G-House Pirates, who are located in Downtown Brooklyn in New York City. The students and the mentors acted like a close family, and by the end of the season, I too felt like I had joined the family. So then, onto the exciting part...

The Robot, Ms Calculated

Our robot for the 2019 FRC season, Ms Calculated, was purpose-built to play FIRST: Destination Deep Space, presented by the Boeing Company. This robot is one of the most complex that the team has built with the current group of students and mentors. Again, we only had six weeks to build this, I am so amazed at what the students accomplished, because the mentors only stepped in to offer advice. Everything was designed and built by the students.

Ms Calculated had sensors and cameras to aid the drivers in collecting Hatch Panels and Cargo from the field’s Loading Station

Using a two stage cascaded elevator design, along with a multi-purpose manipulator for collecting and depositing games pieces, Ms Calculated could score anywhere on the field, including at the Cargo Ship and all three levels of the Rocket with both Hatch Panels and Cargo balls.

Robot Subsystems

Our robot was very complex this year and involved many different moving pieces and subsystems, read more about them below!

Elevator Mechanism

This was the first year our group of students and mentors attempted an elevator, and let’s just say that Ms Calculated could lift the game pieces to outer space! 🚀…or at least to the top of field’s Rockets! It featured the following:

A two stage cascaded elevator with Motion Magic positioning using Talon SRX speed controllers providing pin-point precision for lifting the manipulator to specific predetermined heights with little operator intervention
A second stage containing the mechanism for manipulation field pieces
Dual drive ropes for both lifting and lowering the stages, increasing the speed of the elevator
Mounted Constant Force Springs on each stage allowing for less drive rope tension and required torque to lift the stages

Cargo Intake

The cargo manipulator was placed on the elevator and provided the ability to score anywhere on the field and at any Rocket level.

Collection roller covered with a rubber coating allowed for collecting, holding, and depositing the Cargo
Rotating & locking wrist allowed for both Loading Station and ground collection of Cargo

Hatch Intake

Hatch mechanism extended over the bumpers using drawer-slides driven by pneumatics
A pair of notched poly-carbonate pieces would extend once inside the Hatch Panel, holding it in place
Later iterations, such as automatic collection using Limit Switches & hinged collection allowed for more driver error and faster scoring cycles

Sandstorm / Autonomous Period

The first 15 seconds of every match were played in Sandstorm, where black sheets were covering the Driver Stations, blocking any view of the robot(s) for the drivers, operators, and coaches. To account for this, Ms Calculated had the following hardware on board:

A Limelight camera which could automatically find and follow the reflective tape above the Cargo Ship and Rocket. Using the outputted data, Ms Calculated could drive to and position herself into the scoring position
A Pixy2 camera which could track and follow the lines in front of the scoring targets
A 170° fish-eye camera which allowed the driver and operator to see what was around the robot to navigate the field

Controls / Driver Station

Not wanting to be outdone by the previous year’s Driver Station, the team decided to machine and build a truly unique and special Driver Station, which contained the following:

The driver station laptop and an external monitor, allowing the driver and operator to organize their cameras and telemetry
Holds for the drive controllers
Handles to make it easier to carry
Custom team number and name engravings
A custom-built LED screen and button board for controlling all mechanisms not associated with driving the robot, such as position selection elevator control using a potentiometer and execute button, hatch & cargo collection and deposition, and more

Check out https://imgur.com/gallery/Vii1UYP for some more images of our drive station!

At the End of the Day

We ended up getting eliminated in the closing playoff matches of each of our competitions, but at the end of the day, we still had an amazing year with a great robot and an even greater group of students...here's to 2020 being even better!

Oh, you think this is really cool and want to get involved? Lucky for you, there is probably a team in your area, so message me and I'll make sure to get you involved!... Especially if you're in the NYC area, we could always use more mentors!

Even the Big Ones Mess Up

Daniel Starner — Thu, 27 Dec 2018 19:47:51 +0000

This morning, my feed blew up due to my friends and family complaining about some new Instagram overhaul. Apparently, their feeds were scrolling horizontally instead of the usual vertically. It turns out that this was just one big Oops on the part of Instagram's engineering team, as tweeted by their Head of Engineering.

// Detect dark theme var iframe = document.getElementById('tweet-1078321786586951680-785'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1078321786586951680&theme=dark" }

When stuff like this happens, it's funny to see how quickly the reactions both for and against the change propagate across social media, but in another way, it makes me feel better about being a developer.

If They Can Mess Up, so Can You

Stories like watching Instagram accidentally over-scale their testing of the new UI, or Amazon's Alexa crashing on Christmas due to the influx of new devices makes me realize that no matter how big or powerful these companies are, they are still run by humans, and humans miscalculate and make mistakes.

So if Instagram or Amazon can make these mistakes, why do I give myself so much trouble for writing buggy code sometimes? No one can see all the use cases and outcomes of running their software, and mistakes do happen.

You, me, Amazon or Instagram...we will never write perfect software or always get things right, because there is no right way or perfect software. Whatever works for you, your team, or your company at the time is good enough until you have to make modifications for new user/edge cases.

If we as developers kept programming until we thought our code was "perfect" then either it wouldn't really be perfect, or we'd never finish it! Design and plan ahead as best as possible, but don't beat yourself up for writing buggy code, because it's natural and just happens. If there wasn't any buggy code or systems to fix, a lot of engineers would be out of jobs 🙈

No matter how much we prepare, we will make mistakes, that's just a part of life. What matters is how we face those mistakes and issues, and the tenacity we bring to making software better. These ideas scale, whether we are just solo developers, or are the big companies like Instagram.

So what are your thoughts? How much fuss should we give companies this big when they make mistakes? What are your thoughts on writing code that walks the line between being good and being good enough? What are your thoughts on the balance between tech debt and time to release?

Lessons Learned: Python to JavaScript

Daniel Starner — Wed, 12 Dec 2018 18:01:24 +0000

The following is just some things that I found interesting as I dive deeper into the world of JavaScript.

Some Background

At Bloomberg, I work with a DevOps / Infrastructure team that deals very heavily in hardware, networking, and scripting. This means that everyone on my team is very good with Bash, Ansible, and Python, and can probably rattle off more networking acronyms than seconds in a day.

Shortly after I joined, we started to think about a web dashboard that would manage and automate many of our manual processes and tasks. We were planning all the features to include, and it was going to be so cool...except that I was the only one on the team who knew anything about web development, and even that was a loose association.

My previous experiences were writing backend services and APIs for web companies, but I really had no experience on the front end side of things. For this web dashboard project, I wanted to really dive in and learn React the right way, building out a full application on my own.

What I Learned

Below are just some things that I learned or find interesting about JavaScript, React, and front end development coming from a Python and backend standpoint. Note that I am building a React app, so many of these things revolve heavily around the library.

NPM Dev Dependencies vs Dependencies

I didn't really understand this until I started building Docker images for my Express-React app. In Python, I would always have a dev-requirements.txt file that contained my test and linting libraries and a requirements.txt that held all other dependencies. Its really nice that both of these stay in package.json, cleaning up the project structure and making it easier to automate installs.

Structure Doesn't Matter as Long as It Works

When developing, I am constantly moving files and folders around, trying to achieve the best project structure. I always have the mindset of Will this work if I add more stuff?, and it usually leads to a never ending rabbit hole of project structure management instead of coding.

What I learned from different posts online, and from my own experience, React doesn't care what project structure you use, and so neither should you. Obviously, try to keep it clean and organized, but aside from that, if something works for you, don't bother refactoring it until you have to. I am a fan of my current folder structure that looks something like the following. Note that I omitted files.

.
├── __mocks__
├── bin                # Scripts needed to run Docker image
├── certs              # Any certs I need for local dev. These are mounted to container
├── dist               # Build directory
├── screenshots
├── src
│   ├── assets         # Non-JS or styling assets
│   │   ├── images
│   │   └── jss        # I use MaterialUI, which styles using JSS
│   ├── components     # General use components
│   │   └── hoc        # Higher order components
│   ├── config         # Configuration file for server that loads env to object
│   ├── core           # Anything that is crucial to the React app
│   │   ├── layout     # Core layout components
│   │   │   ├── header
│   │   │   └── sidebar
│   │   ├── pages      # Not Found / Landing pages
│   │   ├── state      # Core managed state, aka users and theme
│   │   │   ├── auth
│   │   │   └── meta
│   │   └── util       # Any usable themes
│   ├── pages          # List of top level React Router routes as 'pages'
│   ├── server         # All code pertaining to the Express server
│   └── tests          # All tests
└── webpack            # Webpack configs for server and client

State Management / Cleaner Components

State management in Python for me doesn't really exist, especially if its for more scripting purposes. I usually favor composition over inheritance, but it was always because of what I learned in school. Writing React components really made this idea stand out.

Components are composed of smaller, possibly reusable components that each level in the hierarchy is responsible for rendering and/or maintaining a specific part of the application. It's a really cool feeling to reduce code lines because I recognized clever ways that components were either related, or could be composed of one another.

The whole idea of the waterfall effect with props and state from parent to children components is really cool to see live, once you understand what is going on. This was something I didn't understand at first, but my code and relationships between components got much better as I understood proper state management.

Promises Are Super Confusing at First

Coming from a synchronous Python / scripting world, JavaScript promises made zero sense to me until about 3 days ago, so don't hate if my examples below are still bad. For the longest time I tried to make Promises synchronous, and I would be so confused at why things like the following returned a promise.

function fetchSomething() {
  const fetchURL = '/something';
  return axios.get(fetchURL);  // returns a promise
}

// Handling the axios call like a synchronous
// function leds to tons of horrible callback 
// and uncaught promise exceptions 🤷🏼‍♂️
function getSomethingHandler(callback) {
  fetchSomething()
    .then(response => { callback(response.data) })
}

But now I understand the whole then(callback), catch(errCallback) flow, and it makes so much more sense. On the next iteration, I rewrote it as the following, which is a bit better:

function fetchSomething() {
  const fetchURL = '/something';
  return new Promise((resolve, reject) => {
    axios.get(fetchURL)
      .then(response => { resolve(response.data) })
      .catch(error => { reject(error) })
  });
}

// Handling the axios call like a synchronous
// function leds to tons of horrible callback 
// and uncaught promise exceptions 🤷🏼‍♂️
function getSomethingHandler(callback) {
  fetchSomething()
    .then(data => { res.send(data) }) })
}

This version stopped many of those callback and promise exceptions. This also allowed my handler functions to expect the data already marshaled into my desired format by the fetch-ing functions. Finally, I started using async and await. I'm still iffy on these, so I apologize if the following isn't 100% correct.

async function fetchSomething() {
  const fetchURL = '/something';
  try {
    // Wait for the axios promise to resolve
    const response = await axios.get(fetchURL);
    return response.data;
  } catch (err) {
    console.log(err)
    return err;
  }
}

function getSomethingHandler(callback) {
  fetchSomething()
    .then(data => { res.send(data) }) })
}

I'm still actively learning more about this asynchronous workflow, but so far its pretty awesome.

Bonus CSS: FlexBox is Amazing

More of a CSS thing, but one of the major factors that kept me from getting into front end design sooner was dealing with element alignment and sizing. No matter what margins or align values I put, nothing seemed to work. This was my first real experience playing with Flexbox in CSS3, and OH SNAP it makes a world of a difference. Between flex-grow, flex-basis, justify, and align-items, positioning things in HTML is made a whole lot easier.

Closing Up

Although there's a lot more stuff I could talk about, these were some of the more major experiences I've had with JavaScript, Express, and React over the past few months. I might write a second post in the future as my project matures.

Thanks for reading 👍😃

DEV Community: Daniel Starner

Serve Static Internal Documentation Behind OAuth Authentication

Background

Maintaining Quality Documentation

Daniel Starner ・ Sep 9 '21

Solution

Tools & Concepts

What is Oauth?

The Process

dstarner / oauth-gated-docusaurus

Running Docusaurus gated behind an OAuth provider session

Creating the Docusaurus Project

Docs-Only Mode

Deploy Basic Version to Heroku

Creating the GitHub OAuth Credentials

Generate a Client Secret

Adding OAuth Proxy to Docusaurus

Testing Login

Bonus: Restricting to a GitHub Organization

Conclusion

Managing Django Media & Static Files on Heroku with Bucketeer

What are Media & Static Files

Django with Media & Static Files

Heroku with Media & Static Files

Complete Example Codebase

dstarner / django-heroku-static-file-example

Used in my blog post of detailing private & public static files for a Heroku-served Django application

Properly Managing Django Media & Static Files on Heroku Example

Bootstrapping Django on Heroku

Introducing Heroku's Bucketeer Add-On

Including the Bucketeer Add-On

Implementing Our Storage Solution

Installing django-storages

Implementing Static, Public & Private Storage Backends

Configure Settings to Use the Storage Backends

Local Development Serving

Enabling S3 Storage

Using the Private Storage

Building Dynamic Breadcrumbs in NextJS

Static Route Breadcrumbs

Defining the Basic Component

Memoizing Generated Breadcrumbs

Improving Breadcrumb Text Display

Nextjs Dynamic Routes

Nextjs Router: asPath vs pathname

Full Example

Switching Jobs in Tech Industry: 6 Key Lessons I've Learned

Con: Productivity Will Be Tough at First

Grass is Always Greener on the Other Side

Politics Exist Everywhere

One Man's Trash...

Be Open to New Things

Be Confident without Faking It

Don't Stress About Proving Yourself

Pro: New Colleagues

Initiate 1:1s with Everyone Around You

Bonus: Write Down Notes for EVERYTHING

Introspecting Python Parameter Values via Argument Binding

Functions Review

Positional Arguments

Keyword Arguments

Default Keyword Arguments

Problem Statement

Python's Signature Class

Quick Look at Decorators

Signatures and Binding

Real World Example

Maintaining Quality Documentation

The Curse of Success

Tribal Knowledge and the Silo of Doom

Cue the Documentation Attempts

Documenting the Web Interface

elrumordelaluz / reactour

Tourist Guide into your React Components

Documenting the RESTful API

axnsan12 / drf-yasg

Automated generation of real Swagger/OpenAPI 2.0 schemas from Django REST Framework code.

Documenting the CLI

spf13 / cobra

A Commander for modern Go CLI interactions

Installing `django-storages`

Nextjs Router: `asPath` vs `pathname`