DEV Community: Lukas Krimphove

Deploying Your Outdoor Activities Map with Terraform

Lukas Krimphove — Thu, 14 Sep 2023 12:49:12 +0000

Introduction

In my last article, I showed you how you can use Python and Folium to create an interactive app to showcase all your outdoor activities:

Visualizing Outdoor Activities with Python Folium

Lukas Krimphove ・ Sep 4

#python #showdev #tutorial #datascience

Now you've created a captivating map that brings your outdoor activities to life. You can use those maps to visually retrace your journeys, celebrate your progress, and relive those adventures.

But what good is it if only you can see it?

Are you ready to showcase your adventures to the world? In this next story, I'll walk you through the process of deploying your interactive map to AWS using Terraform. This tool is great for managing infrastructure as code, and we'll go through the necessary steps to provision and configure the resources needed for your map to be accessible to anyone with an internet connection. By utilizing S3 buckets, Lambda functions, and CloudFront, we'll create a solution that is simple to deploy and maintain.

So get ready to share your outdoor accomplishments with loved ones and fellow outdoor enthusiasts all across the globe!

The Solution

As you can see our solution consists of multiple elements:

At its core, we have two S3 buckets, one for input and another for output. The input bucket becomes the repository for your GPX files, where you'll store the raw material of your outdoor activities. Whenever a new GPX file is uploaded, an EventBridge event is triggered, signaling the arrival of fresh data.
This is where the Lambda function steps onto the stage. It takes the newly arrived GPX file, parses it to extract the essential trail data, and plots the trails on the map. The Lambda function then creates the HTML of the map and places it into the output bucket, ready to be shared with the world.
To guarantee swift and seamless access to the map, we use CloudFront. This service acts as a content distribution network that caches and delivers the output buckets content to users globally from edge locations. It reduces latency and improves performance.

What is Terraform?

Terraform is an open-source infrastructure-as-code software tool created by HashiCorp. It allows you to define and manage your infrastructure as code, making it easy to provision and manage resources across multiple cloud providers. With Terraform, you can ensure consistent and repeatable deployments, making it an ideal choice for automating your cloud infrastructure.

Setting Up the Environment

Before we begin, ensure you have Terraform installed on your local machine. You can download it from the official website and follow the installation instructions.

You will also have to install the AWS CLI and have to configure it so that you can deploy to your AWS account. I wrote a story on using the Windows Subsystem on this.

Once you have Terraform installed, create a new directory for your project and place the main.tf file provided above in this directory. This file contains the Terraform configuration that describes the resources we need to deploy the map.

Provisioning AWS Resources

Our map requires several AWS resources to be provisioned, such as S3 buckets for storing the website files, an AWS Lambda function to generate the map, and a CloudFront distribution to serve the map securely and efficiently.

Buckets

We'll use S3 buckets to store the website files and the map generated by the Lambda function. The provided Terraform code uses the terraform-aws-modules/s3-bucket module to create the buckets.

data "http" "mime_types" {
  url = "https://gist.githubusercontent.com/lkrimphove/46988dc2ac63ad5ad9c95e6109e3c37e/raw/2349abeb136f1f8dbe91c661c928a5ce859432f9/mime.json"
  request_headers = {
    Accept = "application/json"
  }
}

locals {
  mime_types = jsondecode(data.http.mime_types.response_body)
}



### BUCKETS

module "input_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"

  bucket = var.input_bucket
  acl    = "private"

  control_object_ownership = true
  object_ownership         = "ObjectWriter"

}

module "output_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"

  bucket = var.output_bucket
  acl    = "private"

  control_object_ownership = true
  object_ownership         = "ObjectWriter"
}

resource "aws_s3_object" "object" {
  for_each     = fileset("../src/website", "*")
  bucket       = module.output_bucket.s3_bucket_id
  key          = each.value
  acl          = "private"
  source       = "../src/website/${each.value}"
  content_type = lookup(local.mime_types, split(".", each.value)[1], null)
  etag         = filemd5("../src/website/${each.value}")
}

Lambda Function

The Lambda function is the heart of our map generation process. It takes the GPX data from the S3 bucket, processes it, and generates an interactive map. We'll use the terraform-aws-modules/lambda module to create and manage the Lambda function.

### LAMBDA

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "outdoor-activities-generator"
  description   = "Generates a map containing your outdoor activities"
  handler       = "main.lambda_handler"
  runtime       = "python3.11"
  timeout       = 60

  source_path = "../src/lambda"

  environment_variables = {
    START_LATITUDE             = var.start_latitude
    START_LONGITUDE            = var.start_longitude
    ZOOM_START                 = var.zoom_start
    INPUT_BUCKET               = module.input_bucket.s3_bucket_id
    OUTPUT_BUCKET              = module.output_bucket.s3_bucket_id
    S3_OBJECT_NAME             = "map.html"
    CLOUDFRONT_DISTRIBUTION_ID = module.cloudfront.cloudfront_distribution_id
  }

  layers = [
    module.lambda_layer.lambda_layer_arn,
  ]

  attach_policy = true
  policy        = aws_iam_policy.lambda_policy.arn
}

module "lambda_layer" {
  source = "terraform-aws-modules/lambda/aws"

  create_function = false
  create_layer    = true

  layer_name          = "outdoor-activities-layer"
  description         = "Lambda layer containing everything for Outdoor Activities"
  compatible_runtimes = ["python3.11"]
  runtime             = "python3.11" 

  source_path = [
    {
      path             = "../src/lambda-layer"
      pip_requirements = true
      prefix_in_zip    = "python" # required to get the path correct
    }
  ]
}

resource "aws_iam_policy" "lambda_policy" {
  name = "outdoor-activities-generator-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "s3:GetObject"
        Effect   = "Allow"
        Resource = "${module.input_bucket.s3_bucket_arn}/*"
      },
      {
        Action   = "s3:ListBucket"
        Effect   = "Allow"
        Resource = module.input_bucket.s3_bucket_arn
      },
      {
        Action   = "s3:PutObject"
        Effect   = "Allow"
        Resource = "${module.output_bucket.s3_bucket_arn}/*"
      },
      {
        Action   = "cloudfront:GetDistribution"
        Effect   = "Allow"
        Resource = module.cloudfront.cloudfront_distribution_arn
      },
      {
        Action   = "cloudfront:CreateInvalidation"
        Effect   = "Allow"
        Resource = module.cloudfront.cloudfront_distribution_arn
      }
    ]
  })
}

resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = module.lambda_function.lambda_function_arn
  principal     = "s3.amazonaws.com"
  source_arn    = module.input_bucket.s3_bucket_arn
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = module.input_bucket.s3_bucket_id

  lambda_function {
    lambda_function_arn = module.lambda_function.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.allow_bucket]
}

CloudFront Distribution

To serve the map with low latency and high performance, we'll use CloudFront, AWS's content delivery network (CDN). CloudFront caches the map in edge locations worldwide, reducing the load on the origin server (our S3 bucket). We'll use the terraform-aws-modules/cloudfront module to create the CloudFront distribution.

### CLOUDFRONT

module "cloudfront" {
  source              = "terraform-aws-modules/cloudfront/aws"
  comment             = "Outdoor Activities Cloudfront"
  is_ipv6_enabled     = true
  price_class         = "PriceClass_100"
  wait_for_deployment = false

  create_origin_access_identity = true
  origin_access_identities = {
    s3_bucket = "s3_bucket_access"
  }

  origin = {
    s3_bucket = {
      domain_name = module.output_bucket.s3_bucket_bucket_regional_domain_name
      s3_origin_config = {
        origin_access_identity = "s3_bucket"
      }
    }
  }

  default_cache_behavior = {
    target_origin_id       = "s3_bucket"
    viewer_protocol_policy = "redirect-to-https"

    default_ttl = 5400
    min_ttl     = 3600
    max_ttl     = 7200

    allowed_methods = ["GET", "HEAD"]
    cached_methods  = ["GET", "HEAD"]
    compress        = true
    query_string    = false

    function_association = {
      viewer-request = {
        function_arn = aws_cloudfront_function.viewer_request.arn
      }
    }
  }

  default_root_object = "index.html"

  custom_error_response = [
    {
      error_code         = 403
      response_code      = 404
      response_page_path = "/404.html"
    },
    {
      error_code         = 404
      response_code      = 404
      response_page_path = "/404.html"
    }
  ]
}

data "aws_iam_policy_document" "s3_policy" {
  version = "2012-10-17"
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${module.output_bucket.s3_bucket_arn}/*"]
    principals {
      type        = "AWS"
      identifiers = module.cloudfront.cloudfront_origin_access_identity_iam_arns
    }
  }
}

resource "aws_s3_bucket_policy" "docs" {
  bucket = module.output_bucket.s3_bucket_id
  policy = data.aws_iam_policy_document.s3_policy.json
}

resource "aws_cloudfront_function" "viewer_request" {
  name    = "cloudfront-viewer-request"
  runtime = "cloudfront-js-1.0"
  publish = true
  code    = file("../src/viewer-request.js")
}

Deploying Your Map

Create a deploy.tfvars.json file and change the values to fit your map (you have to change the bucket names, as those have to be globally unique):

{
    "input_bucket": "outdoor-activities-input",
    "output_bucket": "outdoor-activities-output",
    "start_latitude": "48.13743",
    "start_longitude": "11.57549",
    "zoom_start": "10"
}

Create a output.tf file (this will print out information to the console):

output "cloudfront_distribution_domain_name" {
    value = module.cloudfront.cloudfront_distribution_domain_name
}

Once you've set up the Terraform environment and configured the main.tf and deploy.tfvar.json files, run the following commands in your terminal:

Initialize Terraform:
terraform init
Plan the deployment to see what resources will be created:
terraform plan -var-file=„deploy.tfvars.json“
Apply the changes to provision the resources:
terraform apply -var-file=„deploy.tfvars.json“

Terraform will show you a summary of the changes that will be made. If everything looks good, type yes to apply the changes. Terraform will now create all the necessary AWS resources for your map. You will find your URL in the console.

Now you are ready to upload your GPX files to the input bucket. Make sure to keep this file structure:

    input-bucket
    ├── Hiking
    │   ├── Trail Group 1
    │   │   ├── Activity_1.gpx
    │   │   ├── Activity_2.gpx
    │   │   └── ...
    │   └── Trail Group 2
    │       ├── Activity_1.gpx
    │       ├── Activity_2.gpx
    │       └── ...
    ├── ...
    └── Skiing
        ├── Trail Group 1
        │   ├── Activity_12.gpx
        │   ├── Activity_13.gpx
        │   └── ...
        └── Trail Group 3
            ├── Activity_14.gpx
            ├── Activity_15.gpx
            └── ...

Conclusion

Congratulations! You've successfully deployed your interactive map of outdoor activities using Terraform and AWS. Your map is now accessible to the world, allowing others to explore your exciting adventures and celebrate your progress.

With Terraform's infrastructure-as-code approach, you can easily manage and update your map in the future. You can add new activities by simple uploading new gpx files to the input bucket. If you want to modify the map's appearance, or enhance it with additional features, you can do all this with just a few changes to the Terraform configuration.

So go ahead, share your map with friends, family, and fellow outdoor enthusiasts.

Happy mapping!

What's Next?

You've learned the basics of deploying your map with Terraform. But there's so much more you can do to enhance your map and create an even richer experience for your audience:

Add your own domain to your CloudFront distribution for easier access.
Don't want to share your map with everybody? Control access to your map by adding authentication and authorization features with AWS Cognito.
Set up a continuous deployment pipeline to automatically update your map whenever you push code changes to your git-repo.

The possibilities are endless. Have fun exploring and expanding your outdoor activities map!

References

This article was originally published in "Python in Plain English" on Medium.

Visualizing Outdoor Activities with Python Folium

Lukas Krimphove — Mon, 04 Sep 2023 04:00:00 +0000

I dream of hiking from Munich to Venice, trekking across the beautiful Alps. But as I still have to live my everyday life, my journey has to consist of multiple stages, with weeks, months, or even years between each adventure. That is ok, as it’s more about the journey than the destination. However, I always wanted a way to visually retrace these paths, to see how far I’ve come and how close I am to my goal. I wanted a way to celebrate the progress I’ve made and to motivate myself to take the journey further.

Fortunately, many outdoor and sports apps, like Adidas Running, Komoot, Strava, and others, graciously allow us to export our activities as GPX files. However, there’s nowhere to go with those GPX files.

That is where Python and Folium come into play. I recently stumbled upon Folium, a powerful Python library for creating interactive maps. It can easily incorporate geographic data, such as the GPX files, and allows for customization and exploration. Drawing upon the wealth of GPS data I’ve gathered, I started experimenting with Folium to craft a map that brings my outdoor excursions to life.

After some research and a lot of testing, I came up with a map that allows me to revisit my past outdoor activities:

So, if you are anything like me and have a treasure trove of GPS data without any purpose, you may wonder how I got there. So in this story, I will explain how to breathe life into your GPX files. Let’s embark on an expedition of exploration and mapping!

Getting Started with Jupyter Notebook

To begin this adventure, we’ll be using Jupyter Notebook. Why Jupyter Notebook? It’s a fantastic interactive computing environment that allows us to combine code, visualizations, and text, making it perfect for experimenting with our data and the Folium library.

If you haven’t installed Jupyter Notebook yet, follow the instructions on their official website. Once installed, you can create a new Jupyter Notebook and get ready for your journey.

Parsing GPX Files for Trail Data

Next, we need the raw material for our map — the GPX files. GPX (GPS Exchange Format) is a widely-used file format that stores location data, such as latitude, longitude, elevation, and time, making it ideal for tracking outdoor activities.

If you’re an avid hiker, runner, cyclist, or skier, chances are you already have tracked various excursions using an outdoor or sports app. Plenty of those apps allow you to export your activities in GPX format. So gather those GPX files, and let’s get started!

In our Python code, we’ll use the gpxpy library to parse the GPX files and extract the essential trail data, such as latitude, longitude, elevation, speed, and distance. The parse_gpx() function will do all the heavy lifting for us:

### READ GPX FILES

import gpxpy
import pandas as pd
import numpy as np
import haversine as hs
from pathlib import Path


def parse_gpx(file_path):
    # parse gpx file to pandas dataframe
    gpx = gpxpy.parse(open(file_path), version='1.0')

    data = []
    points = []
    for track in gpx.tracks:
        for segment in track.segments:
            for point_idx, point in enumerate(segment.points):
                points.append(tuple([point.latitude, point.longitude]))

                # calculate distances between points
                if point_idx == 0:
                    distance = np.nan
                else:
                    distance = hs.haversine(
                        point1=points[point_idx-1],
                        point2=points[point_idx],
                        unit=hs.Unit.METERS
                    )

                data.append([point.longitude, point.latitude,point.elevation, point.time, segment.get_speed(point_idx), distance])

    columns = ['Longitude', 'Latitude', 'Elevation', 'Time', 'Speed', 'Distance']
    gpx_df = pd.DataFrame(data, columns=columns)

    return points, gpx_df


activities = {}
frames = []
for activity_type in ACTIVITY_TYPES:
    activities[activity_type] = {}

    pathlist = Path(activity_type).glob('**/*.gpx')
    for path in pathlist:
        # exclude hidden directories and files
        # will lead to performance issues if there are lots of hidden files
        if any(part.startswith('.') for part in path.parts):
            continue

        activity_group = path.parts[1]
        activity_name = path.parts[2]

        if activity_group not in activities[activity_type]:
            activities[activity_type][activity_group] = []

        points, gpx_df = parse_gpx(path)

        gpx_df['Elevation_Diff'] = np.round(gpx_df['Elevation'].diff(), 2)
        gpx_df['Cum_Elevation'] = np.round(gpx_df['Elevation_Diff'].cumsum(), 2)
        gpx_df['Cum_Distance'] = np.round(gpx_df['Distance'].cumsum(), 2)
        gpx_df['Gradient'] = np.round(gpx_df['Elevation_Diff'] / gpx_df['Distance'] * 100, 1)

        activities[activity_type][activity_group].append({
            'name': activity_name.replace('.gpx', '').replace('_', ' '),
            'points': points,
            'gpx_df': gpx_df
        })
        frames.append(gpx_df)

df = pd.concat(frames)
df

That leaves us with all the necessary data of an activity: a list of all the GPS coordinates and a Pandas DataFrame containing all kinds of metrics.

Plotting GPX Trails on the Interactive Map

With our GPS data now organized in a Pandas DataFrame, we can visualize our outdoor activities on an interactive map. Folium makes this task a breeze:

### CONFIG

# LOCATION = None
LOCATION = [48.13743, 11.57549] # latitude, longitude
ZOOM_START = 10

ACTIVITY_TYPES = {
    'Hiking': {
        'icon': 'person-hiking',
        'color': 'green'
    },
    'Running': {
        'icon': 'person-running',
        'color': 'orange'
    },
    'Biking': {
        'icon': 'person-biking',
        'color': 'red'
    },
    'Skiing': {
        'icon': 'person-skiing',
        'color': 'blue'
    }
}

We’ll create a map centered at a specific location (you can choose one or let the code determine the center based on your data).
We’ll assign unique colors and icons to each activity type, such as hiking, running, biking, or skiing. The ACTIVITY_TYPES dictionary will help us with this.

### CREATE MAP

import folium
from folium import plugins as folium_plugins

if LOCATION:
    location = LOCATION
else:
    location=[df.Latitude.mean(), df.Longitude.mean()]

map = folium.Map(location=location, zoom_start=ZOOM_START, tiles=None)
folium.TileLayer('OpenStreetMap', name='OpenStreet Map').add_to(map)
folium.TileLayer('Stamen Terrain', name='Stamen Terrain').add_to(map)


### MAP TRAILS

def timedelta_formatter(td):
    td_sec = td.seconds
    hour_count, rem = divmod(td_sec, 3600)
    hour_count += td.days * 24
    minute_count, second_count = divmod(rem, 60)
    return f'{hour_count}h, {minute_count}min, {second_count}s'

def create_activity_popup(activity):
    df = activity['gpx_df']
    attributes = {
        'Date': {
            'value': df['Time'][df.index[0]].strftime("%m/%d/%Y"),
            'icon': 'calendar'
        },
        'Start': {
            'value': df['Time'][df.index[0]].strftime("%H:%M:%S"),
            'icon': 'clock'
        },
        'End': {
            'value': df['Time'][df.index[-1]].strftime("%H:%M:%S"),
            'icon': 'flag-checkered'
        },
        'Duration': {
            'value': timedelta_formatter(df['Time'][df.index[-1]]-df['Time'][df.index[0]]),
            'icon': 'stopwatch'
        },
        'Distance': {
            'value': f"{np.round(df['Cum_Distance'][df.index[-1]] / 1000, 2)} km",
            'icon': 'arrows-left-right'
        },
        'Average Speed': {
            'value': f'{np.round(df.Speed.mean() * 3.6, 2)} km/h',
            'icon': 'gauge-high'
        },
        'Max. Elevation': {
            'value': f'{np.round(df.Elevation.max(), 2)} m',
            'icon': 'mountain'
        },
        'Uphill': {
            'value': f"{np.round(df[df['Elevation_Diff']>0]['Elevation_Diff'].sum(), 2)} m",
            'icon': 'arrow-trend-up'
        },
        'Downhill': {
            'value': f"{np.round(abs(df[df['Elevation_Diff']<0]['Elevation_Diff'].sum()), 2)} m",
            'icon': 'arrow-trend-down'
        },
    }
    html = f"<h4>{activity['name'].upper()}</h4>"
    for attribute in attributes:
        html += f'<i class="fa-solid fa-{attributes[attribute]["icon"]}" title="{attribute}">  {attributes[attribute]["value"]}</i></br>'
    return folium.Popup(html, max_width=300)

feature_groups = {}
for activity_type in activities:
    color = ACTIVITY_TYPES[activity_type]['color']
    icon = ACTIVITY_TYPES[activity_type]['icon']

    for activity_group in activities[activity_type]:      
        # create and store feature groups
        # this allows different activity types in the same feature group
        if activity_group not in feature_groups:
            # create new feature group
            fg = folium.FeatureGroup(name=activity_group, show=True)
            feature_groups[activity_group] = fg
            map.add_child(fg)
        else:
            # use existing
            fg = feature_groups[activity_group]

        for activity in activities[activity_type][activity_group]:
            # create line on map
            points = activity['points']
            line = folium.PolyLine(points, color=color, weight=4.5, opacity=.5)
            fg.add_child(line)

            # create marker
            marker = folium.Marker(points[0], popup=create_activity_popup(activity),
                                   icon=folium.Icon(color=color, icon_color='white', icon=icon, prefix='fa'))
            fg.add_child(marker)

map.add_child(folium.LayerControl(position='bottomright'))
folium_plugins.Fullscreen(position='topright').add_to(map)
map

We’ll use Foliums FeatureGroup concepts to group the trails based on their group name. That will allow us to show and hide certain activity groups later.
Now, we’ll iterate through our parsed data and plot each trail on the map using Folium’s PolyLine and Marker objects. The PolyLine will represent the actual trail, while the Marker will act as a starting point for each activity. When you click on a marker, a popup will display relevant information about the corresponding trail.

Conclusion

In this journey of exploration and mapping, we’ve learned how to use Python and Folium to transform mundane GPX files into a dynamic and interactive map. Now you can relive your outdoor adventures, celebrate your progress, and stay motivated for the next stage of your journey.

However, there are many ways to customize, extend and improve your map. So grab your GPX files, fire up Jupyter Notebook, and let your past outdoor activities come to life on the map!

Happy mapping!

What’s next?

Stay tuned because there is much more to come:

deploying a website with your map using AWS
plotting elevation and speed profiles using Python and Plotly
enhancing trails with pictures taken one the way
and much more

References

All the code, including the Jupyter Notebook, is on my GitHub.
While doing my research on Folium, I found a great story by Patrick, who did the same thing I planned to do. His work was a great base to build upon my solution, so please check it out.
Jupyter Notebook
Folium
Pandas

This article was originally published in "Python in Plain English" on Medium.

The Importance of a Software Engineering Daybook

Lukas Krimphove — Mon, 28 Aug 2023 19:49:01 +0000

In the world of software engineering, staying organized, focused, and continuously improving can be challenging. To tackle this, engineers often use various tools and techniques to streamline their work and increase productivity. One method is the software engineering daybook. This approach, described in topic 22 of "The Pragmatic Programmer," provides a unique way to enhance skills and progress in the software development field.

What is a Daybook?

Andrew Hunt and David Thomas' book, "The Pragmatic Programmer: Your Journey to Mastery," is a reliable source of practical programming knowledge. One of its most noteworthy concepts is the daybook, which is a useful tool for software engineers. The authors suggest maintaining a Daybook to record daily lessons, progress, and reflections, similar to the journaling process used by many electrical or mechanical engineers for years.

What to Capture and How?

Record Your Progress and Challenges

One of the central elements to write about is your daily progress. Document the coding problems you tackle, the solutions you devise, and the roadblocks you encounter. Be honest in sharing your challenges, as this transparency is necessary for constructive self-analysis.

Realizations and Breakthroughs

Don't let those hard-earned "eureka" moments sink into oblivion. Note down insights, innovative workarounds, and fresh perspectives that arise during coding. It's always wise to keep notes as they may prove useful in solving future problems that may arise. You never know when you might face a similar issue.

Reflections and Lessons Learned

Take a moment to reflect on your daily choices. Consider what went well and where improvements can be made. Acknowledge both your successes and setbacks to gain valuable insight and improve decision-making skills.

Code Snippets and Patterns

In addition to narratives, the daybook is also a great place to store code snippets, algorithms, and design patterns that you've found useful. This creates a dynamic code repository within your journal that can be quickly accessed for reference.

Personal Growth and Skill Development

Beyond coding triumphs, touch upon your journey. Have you improved your debugging skills? Have you completed a certification? Celebrate your milestones and outline areas you're actively developing.

Why Keep a Daybook?

"The Pragmatic Programmer" lists three benefits of keeping a daybook:

"It is more reliable than memory." Your daybook acts as a secure vault to keep information.
"It gives you a place to store ideas that aren't immediately relevant to the task at hand." Your daybook preserves ideas unrelated to your current tasks for future reference.
"It acts as a kind of rubber duck." Writing in your daybook can reveal fresh perspectives and clarify complex problems.

Other benefits include:

Structured Learning

A well-maintained daybook acts as a knowledge base
You're cataloging lessons that are invaluable not only to you but also to colleagues who might encounter similar challenges.

Iterative Progress

Looking back on your previous entries can reveal your progress over time. It's an opportunity to assess your development, identify patterns in your coding approach, and improve your strategies.

Strategic Decision-Making

The act of recording your choices forces you to think critically about your actions. This intentional approach helps improve your decision-making skills, resulting in more efficient problem-solving.

Analog vs. Digital?

David Thomas and Andrew Hunt recommend using pen and paper over digital files or wikis. However, I have a different viewpoint. In a previously published Medium story, I shared my personal experience of shifting from using an analog notebook to Obsidian, a digital solution. Each method has its benefits, and the choice ultimately depends on personal preference.

Here are some of the most important differences:

Analog vs. Digital Daybook Comparison

Advantages	Analog Daybooks	Digital (Obsidian) Daybooks
Simplicity and Tactile Experience	The physical nature offers tactile comfort.	Smooth navigation and search features for quick access.
Freedom of Creativity	Directly draw sketches, diagrams, designs.	Embed diagrams, images, code snippets for visual support.
Portability	Independent of power, carry anywhere.	Accessible on devices, convenient for digital users.
Security and Loss	Vulnerable to physical damage, loss.	Backed up, synchronized via cloud services to prevent data loss.
Search and Organization	Manual flipping, indexing, challenging.	Powerful search, tagging for efficient organization.
Linking and Interconnectivity	Linking notes is manual, less seamless.	Easy linking of notes, creating interactive network.
Space and Scalability	Limited space, unwieldy with heavy use.	Expandable almost infinitely for future recordings.
Data Security and Privacy	Can be read by others if found.	Passwords, encryption for privacy protection.
Collaboration	Difficult to use, edit collaboratively.	Enables collaborative note editing for teamwork.

Conclusion

As a software engineer, your daybook is an essential tool for navigating challenges, harnessing knowledge, and fostering personal growth. It represents your dedication to learning and serves as a guide towards achieving engineering excellence.

So do you already have a daybook? If not, you should give it a try!

Also published here.