DEV Community: Nakul Kurane

Backing Up My Logic Pro Projects to AWS S3 (Part 1)

Nakul Kurane — Sun, 23 Feb 2020 16:08:15 +0000

On a whim, just to play with Python and AWS, I thought of building a script to backup my Logic Pro projects to AWS S3. The premise of the script is that it will check a directory on my Macbook and upload any files that have been modified since the last time the file was uploaded to S3. This will run via a cron job (which, yes, only runs if my Macbook is awake). Anyway, let’s get started.

As for how I built it, some prerequisites:

PyCharm or any IDE
Python 3.0+
AWS account (free tier should suffice for testing)
Basic understanding of Python/programming

This post won’t go into every code snippet as you can find it on my GitHub (coming soon), but will walk through the steps I took to assemble the components needed.

Objective

Compare files in a local folder with the respective files in S3 and zip and upload the local file if the local file is later than the one on S3 (last modified date time)

Part 1

In Part 1 of implementing this, we will simply go over how to connect to S3 via Python and how to get the last modified date time for a later comparison.

So, we’ll go through the snippets for these steps and/or you can skip all of this and just check out the code on GitHub (coming soon).

Connecting to S3

How to connect to AWS S3? Fortunately, AWS offers a SDK for Python called boto3, so you simply need to install this module and then import it into your script.

I am using Python 3.7 and pip3 so my command was pip3 install boto3

Import the following:

python

import AWS modules

import boto3
import logging
from botocore.exceptions import ClientError

You also need to install the aws cli module (AWS command line interface) so go ahead and do that with a pip3 install aws cli.

Now, you will need to configure this aws cli tool to connect to your AWS account, so you should have an IAM user and group for the cli. Go to the IAM service in your AWS console to create that.

Creating IAM User and Group for AWS CLI

First, create a Group and attach the AdministratorAccess policy. Then, create a User and assign the User to the Group you just created. Continue to the end of the wizard and eventually, you will see a button to download a CSV which will contain your credentials. These will be the credentials you use to configure your aws cli.

Have the Access Key ID and Secret Access Key ready and in your Terminal, type aws configure. You will then be asked for the Key ID and Access Key so simply paste the values from the CSV. You will also be asked for region name (input the region where your S3 buckets are). The last input field asks for output format (you can just press Enter).

Great! Now, your aws cli should be configured to connect to your AWS S3 bucket(s).

Create S3 Client and Resource

So, next we need to create a client and a resource for accessing S3 methods (the client and resource offer different methods based on usage, you can read more here).

python

CREATE CLIENT AND RESOURCE FOR S3

s3Client = boto3.client('s3')
s3Resource = boto3.resource('s3')

object for all s3 buckets

bucket_name = '' # NAME OF BUCKET GOES HERE, HARD CODED FOR NOW

CREATE BUCKET OBJECT FOR THE BUCKET OF CHOICE

bucket = s3Resource.Bucket(bucket_name)

Retrieve object's last modified date time (upload time)

Ok, so what needs to be done to retrieve an object’s last modified date time?

I’ve written a method to loop through the objects in a bucket and call an AWS method to get the last modified time, shown below (yes, this can be written other ways as well).

The bucket.objects.all() returns a Collection which you can iterate through (read more on Collections here).

So you can see I began looping through that and only calling the last_modified method if the S3 object contained Logic_Projects and .zip in the key name.

If you’re not familiar, the key is simply how S3 identifies an Object. I am looking for Logic_Projects because I made a folder with that name and am also checking for .zip just in case I upload something else in the folder by accident (but this script will only upload zips to that folder, so it’s just a safety check).

If it passes those checks, then I proceed to the date time conversions.

You’ll notice I am calling two other methods in this loop, called utc_to_est and stamp_to_epoch before I finally return a value. That is to make datetime comparisons easier later on. The last_modified method returns a date in UTC format so it could look like this: 2019-12-07 19:47:36+00:00.

When I am comparing modification times, I’d rather just compare numbers. The date AWS returns is in UTC, also referred to as GMT (Greenwich Mean Time).

So I added a function to convert this datetime to Eastern Standard Time, shown below:

Now that we’re in the right timezone, I want to convert it to a number, so I converted it to an Epoch time, which is the number of seconds that have elapsed since 00:00:00 UTC on January 1, 1970 (Why Epoch time is this date is not in the scope of this post).

OK, that's all for Part 1! In this post, we've connected to AWS via Python (boto3) and we've created a method to get the last modified time (in seconds since Epoch and in EST) of an object in S3.

In Part 2, we will go over comparing the last modified time of a file in S3 with the last modified time of that file locally.

This post also appears on Medium (link below).

Backing Up My Logic Pro Projects to AWS S3 (Part 1) | by Nakul Kurane | Analytics Vidhya | Medium

Nakul Kurane ・ Mar 2, 2020 ・
Medium

How to Execute a Cron Job on Mac with Crontab

Nakul Kurane — Mon, 17 Feb 2020 17:17:13 +0000

Prologue

Before I write the steps below, I would recommend installing the latest version of Python with homebrew. Homebrew is a package manager just like pip and anaconda and frankly, I’m just using that in this example because I tried for hours to run a simple cron job to no avail. I frequently got errors about not being able to open the file I wanted to execute or not finding the installed python version. I won’t get into my errors in this post but you’re free to use any other Python installation if you like.

I uninstalled my Python 3.x versions I had installed (via pip and conda) and started from scratch — uninstalling was also a hassle. So let’s say you’re on your Mac and have the default Python distribution which I believe should be 2.7.10. Few to no developers are updating modules in this version so when you’re building projects, it’s best to have the latest version of Python installed.

I followed Real Python’s guide to install Python 3 via Homebrew (a quick few steps really) so I will assume you have done the same going forward in this post.

Steps

Open up your Terminal command prompt on your Mac and navigate to the home directory by running cd ~/

For me, it is Users/Nakul

We will be using Mac OS’s in-built crontab feature to write our cron jobs.

Type crontab -e and press Enter.

This should open up an empty file, this is where you will write your cron jobs. You can write the job to run a shell script or a Python script in this case.

Type :q! to exit the editor.

Assuming you followed the Homebrew guide above, double check your Python installation path. You can do so by typing which python3 in your Terminal. It should display /usr/local/bin/python3 (which is an alias for the actual homebrew installation location but that’s fine, we won’t go into that right now). You will include this path in your cron job syntax, which we’ll get into.

Before we write our cron job, we should have a script which we want to run.

I’ve already created a directory /Documents/Python/cron in my home directory and created a simple script called cron_test.py

Note: this Python script needs to be executable so change the permissions on it to allow for that, I just ran a chmod 777 cron_test.py

The cron_test.py script simply creates a directory with the current date and time as the name.

The script text is as follows if you want to use the same script:

python

!/usr/bin/python3

from datetime import datetime
import os
sysDate = datetime.now()

convert system date to string

curSysDate = (str(sysDate))

parse the date only from the string

curDate = curSysDate[0:10]

parse the hour only from the string

curHour = curSysDate[11:13]

parse the minutes only from the string

curMin = curSysDate[14:16]

concatenate hour and minute with underscore

curTime = curHour + ‘_’ + curMin

val for the folder name

folderName = curDate + ‘_’ + curTime

make a directory

os.mkdir(folderName)

You might wonder what that first line #!/usr/bin/python3 is. That line is referred to as a shebang and how the cron job will interpret the script so in this case it will run the cron_test.py with Python (as if you executed python cron_test.py in your Terminal).

Ok, so we have Python 3.x installed, we have a Python script, now let’s write the cron job to run this!

Let’s say we want to run this script every minute (mostly for the purpose of testing on your end so you can see the results without waiting too long).

The cron job syntax is as follows:

*/1 * * * * cd ~/Documents/Python/cron && /usr/local/bin/python3 cron_test.py >> ~/Documents/Python/cron/cron.txt 2>&1

What does the above mean? Let’s analyze it quickly.

*/1 * * * * simply means that the job will run every minute.

cd ~/Documents/Python/cron && /usr/local/bin/python3 navigates to the directory where the script you want to execute is located and specifies to use Python instead of Bash to execute (since it is a Python script, also remember that we added the shebang within the script, I found that it needed to be in both places to work).

cron_test.py is the script filename.

>> ~/Documents/Python/cron/cron.txt specifies where to output the logs in case the execution of the job has any issues.

2>&1 simply disables email because by default, the cron job will try to send an email but we don’t have an address specified.

Anyway, that should be enough to get you writing and testing your scripts, so go on!

Feel free to browse the web to learn more about cron jobs and crontab. I decided to write this for those who have a fundamental understanding of programming and wanted to write a cron job as soon as possible.

Hope this helped; go write some jobs!
Find me on the web:

Follow on LinkedIn

DEV Community: Nakul Kurane

Backing Up My Logic Pro Projects to AWS S3 (Part 1)

Objective

Part 1

Connecting to S3

import AWS modules

Creating IAM User and Group for AWS CLI

Create S3 Client and Resource

CREATE CLIENT AND RESOURCE FOR S3

object for all s3 buckets

CREATE BUCKET OBJECT FOR THE BUCKET OF CHOICE

Retrieve object's last modified date time (upload time)

Backing Up My Logic Pro Projects to AWS S3 (Part 1) | by Nakul Kurane | Analytics Vidhya | Medium

Nakul Kurane ・ Mar 2, 2020 ・ Medium

How to Execute a Cron Job on Mac with Crontab

Prologue

Steps

!/usr/bin/python3

convert system date to string

parse the date only from the string

parse the hour only from the string

parse the minutes only from the string

concatenate hour and minute with underscore

val for the folder name

make a directory

Nakul Kurane ・ Mar 2, 2020 ・
Medium