Automatic Data Updates for Static Websites

99littlebugs — Tue, 23 Nov 2021 16:03:25 +0000

My Workflow

On Sunday, January 10th, 2020, teachers had just became eligible for the COVID-19 vaccine but finding an appointment was tough. Each NY state location had its own sign-up link, and people were cycling through their list every couple of minutes hoping that availability would open. As a developer, I wondered whether I'd be able to help by creating a single pane of glass that would show availability for many locations at a glance.

I wanted something simple, was excited by static sites and serverless and did some fast research on how I could string these concepts together to create an auto-updating website of vaccine appointment availability. I found some tutorials that showed developers scraping websites and then committing the data back to a git repository and knew that’s what I would try first. I told myself that if I could get the data scraping done on that first night then this was a problem worth more of my time.

I used cheerio for the website scraping and wrote the collected data out to a file. Then I used GitHub actions on a 15 minute cron job schedule to checkout the repo, run the scraper script, and commit the data back. In 24 lines of yaml, and 55 lines of JavaScript, I had an automatic data scraper. I was satisfied with my progress and well on my way.

Submission Category:

DIY Deployments

Code

Full source code, which is linked to the last code commit on January 10th.
Particular attention should be given to /.github/workflows/scrape.yml (link) and scraper.js (link).

Additional Resources / Info

Inspiration for this pattern is listed in the repo's README file.

Ordering photos and videos for a slideshow or montage

99littlebugs — Sun, 06 Jun 2021 16:34:02 +0000

The problem

Ordering photos so that when imported from a Windows PC to an iPad (or other tool) they would be in the right order. The end goal was to import them into a slideshow app already in the required order.

Mistake #1 - renaming in Google Drive

I received the photos via email and had imported them directly to Google Drive using the button in Gmail. I then renamed them with a number prefix so that they would alphabetize to the order that I wanted.

When I was ready to start compiling, I downloaded the whole folder to my computer. But they had the original filenames! After some researching, I found that Gmail automatically used an "Image Link" when importing. This meant that I had renamed the link instead of renaming the actual file.

Discovering Google Colaboratory

There was little to nothing on Google about my specific issue and "Image Links" in general, but I discovered a similar problem for a shared folder. One of the responses there suggested using Google Colaboratory which with these two snippets, worked perfectly.

from google.colab import drive
drive.mount('/gdrive')

!cp -r '/gdrive/My Drive/OriginalFolder/.' '/gdrive/My Drive/NewFolder'

This copied over all the files from OriginalFolder to NewFolder within Google Drive.

Using ExifTool

Now that I had the files on my computer and in the right order, I still needed a way to load them to the iPad with the correct ordering. Some of the photos had the Date Taken file metadata which is what the iPad uses for ordering.

Lots of online resources pointed to using ExifTool for cleaning up that metadata and rjames86's exiftool cheatsheet taught me to set all the file dates to an arbitrary time with this command

exiftool.exe -r "-AllDates=2021:06:05 14:00:00" "C:\Path\To\Folder"

I was optimistic that since all the timestamps were now the same, the iPad would resort to ordering alphabetically by filename. I had been using the Files app for importing from Google Drive and the files still weren't coming in 100% in the right order, though they did now have the expected arbitrary date and time set.

What I really needed was a way to set the timestamps to be ordered in the order of the files on my computer. That would be foolproof for ensuring that they loaded in properly. But how to use ExifTool for that? I thought about scripting it myself but didn't really have the time. Thankfully, I found instructions for using ExifTool with the filesequence variable once the files already all have the same time.

exiftool.exe -r "-AllDates+<0:0:${filesequence}0" "C:\Path\To\Folder"

Voila! This worked great. At this point, I also realized that I could've and should've been using iTunes and a cable to sync a folder to the iPad, rather than waiting for upload, download, and then import from Google Drive.

What I'd do next time

Ordering via renaming locally instead of on Google Drive
Use exiftool for the incremental arbitrary timestamping
Use iTunes and a cable to sync the photos to iPad instead of through the internet

DEV Community: 99littlebugs