DEV Community: Red Headphone

How to Time an Unexpected Bug to Rise on a Specific Month

Red Headphone — Fri, 17 May 2024 12:30:00 +0000

In the world of software development, encountering bugs is an inevitable part of the process. Some bugs are predictable and can be tackled with routine testing, while others surface unexpectedly, causing delays and frustration. Interestingly, there are scenarios where specific bugs may arise due to the peculiarities of date and time handling in programming languages. In this blog post, we'll explore a real-life example involving the pandas library in Python, and how an unexpected bug related to date parsing was triggered by a specific month format.

The Setup

Imagine you're working on a cron job that parses futuristic dates around 2-3 months from today. For example, you're parsing data for upcoming events. Here’s how your code should look like:

import pandas as pd

df = pd.DataFrame([
  {"date": "May-16-2024", "event_type": "...", "event_name": "...", "event_description": "..."},
  {"date": "May-17-2024", "event_type": "...", "event_name": "...", "event_description": "..."},
  {"date": "May-24-2024", "event_type": "...", "event_name": "...", "event_description": "..."},
  {"date": "May-27-2024", "event_type": "...", "event_name": "...", "event_description": "..."},
  {"date": "Jun-05-2024", "event_type": "...", "event_name": "...", "event_description": "..."}
])

df["date"] = pd.to_datetime(df["date"])

The Unexpected Bug

When running the above code, you expect pandas to seamlessly convert your date strings into datetime objects. However, instead, you encounter an error:

ValueError: time data "Jun-05-2024" doesn't match format "%B-%d-%Y", at position 4. You might want to try:
    - passing `format` if your strings have a consistent format;
    - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format;
    - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.

What just happened? The issue lies in how pandas infers the date format based on the first element of your data. The first element "May-16-2024" is interpreted as having a full month name ("%B-%d-%Y"), and pandas expects the same format for the subsequent dates. This works for "May-16-2024" to "May-27-2024" since "May" has the same abbreviated and full form. However, it fails when it encounters "Jun-05-2024" because "Jun" is an abbreviation.

Timing the Bug

We can strategically make this bug rise during the whole month of May. This is due to how pandas infers date formats. If the first date in the dataset falls within May, pandas interprets the format as full month names ("%B-%d-%Y"). Since "May" is the same in both full and abbreviated forms, it continues to parse subsequent May dates without issues. However, as soon as it hits a date with an abbreviated month (e.g., "Jun"), it raises an error.

Handling the Bug

To handle this issue, you can specify the format parameter in pd.to_datetime to correctly parse the dates. While pandas provides a workaround using format='mixed', being explicit about the format can save you from unexpected errors:

df["date"] = pd.to_datetime(df["date"], format='%b-%d-%Y')

By specifying '%b-%d-%Y', you indicate that the month names are abbreviated, ensuring pandas correctly parses the dates.

Conclusion

Timing an unexpected bug to rise on a specific month might sound like an odd goal, but understanding how and why certain bugs appear can help you better manage your data parsing tasks. In this case, knowing how pandas interprets date formats can save you from a lot of headaches.

Always be aware of the format of your date strings and specify the format explicitly when using pd.to_datetime to avoid unexpected errors. This practice will ensure your code runs smoothly regardless of the peculiarities of your dataset.

Happy coding!

Automatically Prefix JIRA Issue ID to Git Commit Messages using Git Hooks

Red Headphone — Fri, 23 Feb 2024 08:44:20 +0000

In software development teams, efficient communication and tracking are paramount. Integrating JIRA, a popular issue tracking tool, with Git commits streamlines this process. In this guide, we'll walk through automating the prefixing of JIRA issue IDs to Git commit messages, ensuring clarity and traceability in your project history.

Step 1: Creating the Prepare Commit Message Hook

To automate the prefixing process, we'll utilize Git hooks, specifically the prepare-commit-msg hook. This hook is triggered before a commit message is finalized. Follow these steps:

Create the Hook File: Navigate to your project's .git/hooks directory and create a new file named prepare-commit-msg.
Make the File Executable: Ensure the file is executable by running chmod +x .git/hooks/prepare-commit-msg in your terminal.
Paste the Python Script: Inside prepare-commit-msg, paste the following Python script:



#!/usr/bin/env python3

import sys
import re
from subprocess import check_output

commit_msg_filepath = sys.argv[1]
branch = check_output(
    ["git", "symbolic-ref", "--short", "HEAD"]
).strip().decode()


regex = "[A-Z]{2,7}-[0-9]{1,5}"
if branch.count("-") >= 2:
    if re.match(regex, branch):
        with open(commit_msg_filepath, "r+") as fh:
            commit_msg = fh.read()
            fh.seek(0, 0)
            fh.write("#%s %s" % (branch, commit_msg))

This script fetches the current branch name, checks if it follows the pattern of a JIRA issue ID, and prefixes the commit message accordingly.

Step 2: Testing the Integration

With the hook in place, every commit made on a branch named after a JIRA issue (eg. TEMP-9999) will automatically have the issue ID prefixed to the commit message.

Conclusion

By automating the prefixing of JIRA issue IDs to Git commit messages, teams can ensure consistency and improve traceability in their project history. This integration enhances collaboration and facilitates smoother project management workflows, so you can focus on writing code.

Scraping Data from Websites using JavaScript: A Beginner's Guide

Red Headphone — Sun, 07 Jan 2024 11:00:16 +0000

Scraping data from websites can be a powerful way to gather information for various purposes, from research to data analysis. While there are various tools and languages for web scraping, JavaScript offers a straightforward approach when scraping data from websites directly in the browser.

Let's take a simple example of scraping a country list from a website using JavaScript. The target website is https://www.iban.com/country-codes, and we'll utilize JavaScript code executed in the browser's Developer Tools console to extract this data.

Understanding the Script

The provided JavaScript code aims to extract a country list table from the mentioned website. It uses basic DOM traversal methods to navigate through the table elements and gather the data into a structured format.

// Select all table rows (tr elements) on the webpage
var all_tr_elements = $("tr");

// Initialize arrays to store keys (column names) and data
var keys = [];
var data = [];

// Loop through each table row
for (var ind = 0; ind < all_tr_elements.length; ind++) {
  var tr = all_tr_elements[ind];
  var tds = tr.children;
  var row = {};

  // Loop through each cell (td element) in the row
  for (var i = 0; i < tds.length; i++) {
    // If it's the first row (header), store the column names as keys
    if (ind == 0) {
      keys.push(tds[i].textContent);
    } else {
      // For subsequent rows, populate the row object with cell values
      row[keys[i]] = tds[i].textContent;
    }
  }

  // If it's not the first row, push the row data to the 'data' array
  if (ind != 0) {
    data.push(row);
  }
}

Step-by-Step Guide

Accessing the Website:
Firstly, navigate to the website containing the data you want to scrape. In this case, visit https://www.iban.com/country-codes using your web browser.

Open Developer Tools:
Open the Developer Tools in your web browser. Typically, you can access this by right-clicking on the webpage, selecting "Inspect" or "Inspect Element."

Navigate to Console:
Once the Developer Tools are open, go to the "Console" tab where you can input and execute JavaScript code.

Execute the Script:
Copy and paste the provided script into the Console tab of the Developer Tools and press "Enter" to execute it.

Retrieve Data:
Once the script runs, type "data" in the console to view the extracted data as an object. For a condensed JSON format, use "console.log(JSON.stringify(data))" in the console. Both forms can be copied directly from the console window for further use or analysis.

Important Notes

Respect Website Policies: Always ensure that you're complying with the website's terms of service and policies. Scraping data from websites might be prohibited or restricted in some cases.

Data Structure: The script provided assumes a specific structure of the webpage's HTML. Changes in the website's layout or structure might require adjustments to the scraping code.

Limitations: Directly scraping data from websites via JavaScript may have limitations due to browser security measures or dynamic content loading. More complex scraping tasks might require server-side solutions or dedicated scraping libraries.

Conclusion

Using JavaScript for web scraping can be a convenient method, particularly for simple data extraction tasks. However, for more extensive or complex scraping needs, it might be worthwhile to explore other scraping tools or programming languages tailored for web scraping. Always use scraping responsibly and in compliance with website policies and legal regulations.

Tuning Out Spotify: A Programmer’s Approach to Music Streaming

Red Headphone — Sat, 14 Oct 2023 07:53:03 +0000

In a world full of boundless musical choices, my exploration as a programmer prompted me to challenge the established norms of music platforms like Spotify. While these platforms provide a vast collection of songs readily available, there were certain aspects that consistently bothered me. The constant interruptions from ads that disturbed my rhythm and the need for a consistent internet connection started to become obstacles that I couldn’t overlook any longer. Yet, beyond these frustrations, there was an intriguing option that caught my attention — an option that aligned with my coding instincts.

Picture this usual scenario: a treasured YouTube playlist, carefully curated over time — a situation many of us can easily understand. Could these playlists be changed into portable MP3s, conveniently residing on our devices? However, the idea of individually downloading and converting each track appeared daunting, a task that could eat up a lot of valuable time. This is where my programming skills came to the rescue.

In this venture, two crucial tools became my allies: Google Colab and Google Drive. The former offered its computing power to execute my code, scouring the digital realm for tracks and carefully downloading them, while the latter kindly gave a home to these musical gems, crafting an organized repository that could be accessed whenever needed. To bring these melodies to life, the elegant open-source VLC Player entered the scene, seamlessly weaving the MP3s into audible experiences. Together, this fusion of Google Colab’s computational finesse, Google Drive’s storage capabilities, and VLC Player’s versatile playback features orchestrated a symphony that highlighted the beautiful blend of code and music in a programmer’s universe.

Starting this musical coding adventure requires careful planning. Here’s the simple sequence that makes up the process of getting your music:

Get Pytube Working: Begin by installing Pytube in Google Colab, the YouTube video downloader. Be cautious about potential issues due to delayed updates. A working fork of Pytube can usually be found in GitHub issues, serving as a useful temporary solution. This demonstrates how open-source collaboration can address challenges and provide a smoother experience.

!pip install pytube
# if PyPI release is outdated 
# !pip install git+https://github.com/pytube/pytube

Sync Google Drive: Effortlessly connect Google Colab and Google Drive for seamless data transfer.

from google.colab import drive
drive.mount('/content/drive')

Bring in the Tunes: Work your magic by downloading tracks and giving life to your collection. Remember, the playlist URL you use should be either public or unlisted for this process to work.

from pytube import YouTube
from pytube import Playlist
from os import system

# if there is already music folder in drive
# system("cd drive/MyDrive && rm -r music && mkdir music") 

counter = 0
playlist_url = "{add your playlist url here}"
playlist = Playlist(playlist_url)

for url in playlist:
  music = YouTube(url).streams.filter(only_audio=True)
  counter+=1
  music.first().download(
    output_path="drive/MyDrive/music/",
    filename_prefix=str(counter).zfill(3)+" ")

And that’s the final touch! Inside your Google Drive, the music folder holds all the songs from your YouTube playlist. This seamless blend of code and creativity is set to echo as you download and enjoy your personally crafted collection using the flexible VLC Player.