DEV Community: Kajigga

Using a Proper Text Editor

Kajigga — Wed, 25 Oct 2017 06:36:56 +0000

Hey, you! Yeah, you. Come on over. We need to have a chat. Let's just get a few things out in the open so we can discuss it like adults.

We need to talk about that text editor you might be using.

You should never open or edit a Python file with a program such as Microsoft Word, TextEdit (Mac), LibreOffice, Pages, or any other page layout program. Just say no! It won't work. You will get frustrated. Your code won't run. It won't work. (Did I say that already?) There are pretty much...let's see... zero reasons to open code of any kind in such a program. You are only asking for trouble.^[1]
Continue Reading ->

Recommended Text Editors (2017)

Kajigga — Wed, 25 Oct 2017 06:34:33 +0000

Looking to try out a new text editor? There are literally hundreds of them out in the wild. When it comes to Python, however, only some are useful.

Here are my 2017 text editor recommendations ranked in from highest to lowest.

Continue Reading →

I have a Editor Confession to Make

Kajigga — Wed, 25 Oct 2017 06:30:48 +0000

I have to make a confession. I am a hardcore Vim addict. I start using it 10 years ago when I worked on servers and needed to edit files without a mouse. I spent hours connected to these system and learned to use Vim on the command line. I can't get away from it now. I have so much muscle memory learned with the keyboard shortcuts that I feel handicapped without them in other programs.

I'm not alone. I'm not the only Vim user out there. Read More...

Easy Exporting CSV Data with Python

Kajigga — Tue, 10 Oct 2017 05:09:05 +0000

This article is a follow-up to Reading CSV Data with Python.

In that article, I described what CSV files are and how to read CSV Data. The flip side to reading data is writing data out to a file. I will summarize again what a CSV file is then go into how to write data in this format.

CSV files are extremely common. I run into them all the time at work when transferring data. At home, a CSV file is almost always an option when exporting a file from the bank or out of Google Sheets.

What is CSV?

CSV = C omma S eparated V alues

Photo by Andrew Furlan / Unsplash

We see them all the time but there seems to be some confusion about what a CSV file really is. When I talk to people about CSV files, I realize that most of us equate them with Microsoft Excel documents. This is probably because Excel is often the default program for opening CSV files.

Here is the thing. A CSV file is simply a text file. That's it, nothing more. The file extension is usually .csv but it doesn't have to be. Any file extension is simply a hint to what is contained in the file and doesn't actually determine or control the content type. The Wikipedia Article about CSV files states that:

"CSV" is not a single, well-defined format.

As a result, in practice the term "CSV" might refer to any file that:

is plain text using a character set such as ASCII, various Unicode character sets

consists of records (typically one record per line), with the records divided into fields separated by delimiters

where every record has the same sequence of fields

Here is an example. I have a file called users.csv sitting on my desktop. In a spreadsheet program (Excel, Google Sheets, LibreOffice, Numbers, etc) this file would look like this.

Those columns and rows are nice and fancy. We think there is something magical going one but it isn't. The contents of this file are really just:

user_id,first_name,last_name,email
234,Kevin,Crommer,example@example.com
235,Carl,Marx,cmarx@example.com

You can open a CSV file with a plaintext editor to see the true contents. It is very similar to viewing the source of a web page and looking at the underlying HTML.

A spreadsheet programs reads the CSV file and displays to the user (you and me) a pretty version that makes it easy to see the columns and rows.

As you can see in my example, A CSV file typically has one row per record. In this case there are 4 columns. The headers of the columns are user_id, first_name, last_name and email. Records will have one or more columns that are usually, not always, separated by a comma.

Enough Review Already, Get to the Writing Data Good Stuff

Ok, ok. Let's learn about how to write CSV data to a file or file-like object.

There are a few essentials points that you will need to remember when writing out CSV data.

Do not try to write CSV without a library.

You have better things to do with your time than to figure out how to handle every possible CSV variation. Believe me, I speak from experience. I will admit to wasting days of time before I discovered the built-in (yes built-in) csv module. Your efforts are better spent getting stuff done.

The `csv` module is Awesome

The CSV Module is built into every modern version of Python.^[1]

This module handles all of the intricacies of reading and writing CSV data. It is crazy fast and just works.

To quote the Python Docs:

The csv module implements classes to read and write tabular data in CSV format. It allows programmers to say, “write this data in the format preferred by Excel,” or “read data from this file which was generated by Excel,” without knowing the precise details of the CSV format used by Excel. Programmers can also describe the CSV formats understood by other applications or define their own special-purpose CSV formats.

Enough Enough already... Get to An Example

Okay, okay. Let's start with an example.

Finish reading this article at http://scriptingwithpython.com/easy-exporting-data-as-a-csv-in-python/

Reading CSV Data with Python

Kajigga — Wed, 04 Oct 2017 19:21:01 +0000

What is CSV?

CSV = C omma S eparated V alues

"CSV" is not a single, well-defined format

As a result, in practice the term "CSV" might refer to any file that:

is plain text using a character set such as ASCII, various Unicode character sets

consists of records (typically one record per line), with the records divided into fields separated by delimiters

where every record has the same sequence of fields

Here is an example. I have a file called users.csv sitting on my desktop. In a spreadsheet program (Excel, Google Sheets, LibreOffice, Numbers, etc) this file would look like this.

Those nice columns and rows are nice and fancy. We think there is something magical going one. However, the contents of this file are really just:

user_id,first_name,last_name,email
234,Kevin,Crommer,example@example.com
235,Carl,Marx,cmarx@example.com

You can open a CSV file with a plaintext editor to see the true contents. It is very similar to viewing the source of a web page and looking at the underlying HTML.

A spreadsheet programs reads the CSV file and displays to the user (you and me) a pretty version that makes it easy to see the columns and rows.

Rules for reading CSV Files

Each row in a CSV file is separated by line breaks.
Columns are separated by a known character. They are almost always comma-separated but I have seen commas, tabs, pipes (|) and other wierd stuff.
If a field contains a comma, it should be surrounded by quotes
- Example:

user_id,nicknames
987,"joebob,joey,jos"

If a field contains single quotes then it should be surrounded by double quotes
- Example:

user_id,last_name
987,"O'Neil"

If a field contains double quotes then it should be surrounded by single quotes

user_id,citation
987,'The Man said,"Who goes there?"'

The moral of the story here is:

Don't try to parse CSV files by manually reading a line of the file, splitting by commas into a list.

There are way too many ways to get it wrong. Instead, use an existing library to do this part. In Python, the csv module is essential. It takes care of reading all of these possible things.

Using the builtin Python `csv` module

Here are the basic steps for reading a CSV file with the builtin csv Python module.

Open a CSV file for reading
Pass the file handler to the CSV reader
Iterate through the rows of the file

I will walk through each one in order, showing how I would read my example file from earlier.

Step 1: Open a csv file for reading

ALl during this example, we are going to use the following CSV data. We will assume it is saved a file called quotes.csv sitting next to our script.

quote,speaker
"Don’t you call me a mindless philosopher, you overweight glob of grease!",C-3PO
"We’re doomed.",C-3PO
"But I was going into Tosche Station to pick up some power converters!",Luke Skywalker
"Help me, Obi-Wan Kenobi. You’re my only hope.",Leia Organa
"An elegant weapon for a more civilized age.",Obi-Wan Kenobi
"I find your lack of faith disturbing.",Darth Vader
"Mos Eisley spaceport. You will never find a more wretched hive of scum and villainy.",Obi-Wan Kenobi
"You don’t need to see his identification…These aren’t the droids you’re looking for.",Obi-Wan Kenobi
"It’s the ship that made the Kessel Run in less than 12 parsecs.",Han Solo
"Sorry about the mess.",Han Solo
"She may not look like much, but she’s got it where it counts, kid.",Han Solo
"Governor Tarkin, I should’ve expected to find you holding Vader’s leash. I recognized your foul stench when I was brought on board.",Leia Organa
"I felt a great disturbance in the Force. As if millions of voices suddenly cried out in terror and were suddenly silenced.",Obi-Wan Kenobi
"I suggest a new strategy, Artoo: Let the Wookiee win.",C-3PO
"Hokey religions and ancient weapons are no match for a good blaster at your side, kid.",Han Solo
"That’s no moon. It’s a space station.",Obi-Wan Kenobi
"Who’s the more foolish? The fool or the fool who follows him?",Obi-Wan Kenobi
"Aren’t you a little short for a stormtrooper?",Leia Organa
"Somebody has to save our skins. Into the garbage chute, flyboy!",Leia Organa
"I got a bad feeling about this.",Han Solo
"When I left you I was but the learner. Now I am the master.",Darth Vader
"If you strike me down I shall become more powerful than you can possibly imagine.",Obi-Wan Kenobi
"It’s not impossible. I used to bullseye womp rats in my T-16 back home, they’re not much bigger than 2 meters.",Luke Skywalker
"Cover me, Porkins!",Biggs Darklighter
"Use the Force, Luke.",Obi-Wan Kenobi
"What an incredible smell you've discovered", Han Solo

Opening a file for reading is pretty simple, but I always use the with command to do it. Doing it this way automatically closes the file. Without it you have to call _file.close() yourself or you risk loosing data or corrupting the file. Python is forgiving but I just don't like taking the risk. Here is the first part.

import csv

with open('./quotes.csv', 'r') as _filehandler:

Step 2: Pass the CSV file handler to the Reader

With the filehandler opened, pass it to the csv reader. At this point you have two ways of reading rows. You can use csv.reader to read the rows as simple lists. This method:

reads every row as a unique object
has no notion of column headers. The first line of the file is just another row of data
gives you indexed access to each row
- i.e. row[0] or row[1]

The second option, and the one I personally use the most, is csv.DictReader. This reader:

reads headers from first row
returns each row as a dictionary where the keys come from first row

If you read it using csv.reader then the example becomes:

import csv

with open('./quotes.csv', 'r') as _filehandler:
    csv_file_reader = csv.reader(_filehandler)

If you read it using csv.DictReader then we get:

import csv

with open('./quotes.csv', 'r') as _filehandler:
    csv_file_reader = csv.DictReader(_filehandler)

There isn't a huge difference there but you get to decide. I personally prefer using dictionaries that remembering the proper index for the column I am looking for.

Step 3: Iterate over each row

The final step is to iterate over each row, doing something with it. The pattern for row in csv_file_reader: is very common. This creates an iterator ^[1]

Our example then becomes:

import csv

with open('./example.csv', 'r') as _filehandler:
    csv_file_reader = csv.DictReader(_filehandler)
    for row in csv_file_reader:
        # Do something here
        print(row['quote'])
        print(row['speaker'])

row in this script is a dictionary. We can access the quote with row['quote'] and the person who is speaking with row['speaker'].

Et voila! There you go! I suspect that this pattern will become the basis of many scripting projects in the future.

I recorded a quick video. In it, I show how to create the file in PyCharm and then run it.

Iterators are pretty awesome when you understand what is happening. They allow for very efficient memory usage with large files, remote data, and cases where you don't know how much information will be processed. In the case of the example given here, it causes the reading to only read the file one row at a time. At no point is the entire CSV file loaded into memory. ↩︎

Create a CSV of Yesterday's Google Calendar Events

Kajigga — Mon, 25 Sep 2017 09:15:00 +0000

Python is great at processing structured data. There are all sorts of libraries that you can use to parse JSON, CSV, XML, etc. There are libraries to parse calendar (.ics) files as well.

A few days ago, I needed to pull a CSV report of all of my calendar events from the day before. This task might need to be done more than once. My scripting radar immediately picked up a ping and I saw a perfect target for a Python script.

Sure, I could copy and paste the list from the Agenda view in Google Calendar itself. That could be ok if I only needed to do this once. However, I might do this again and a script gives me way more flexibility and frees me up to do other things.

What is the Task?

Task: Pull a complete list of all calendar events from Google Calendar and save it in a CSV file. The output CVS file must have the following columns.

title	start_date	end_date	duration
Lunch	2017-09-21 09:00:00-06:00	2017-09-21 09:55:00-06:00	55

Solution to pulling Google Calendar Events to CSV

Pre-requisites

Python 2.7, preferably Python 3.4 or higher see my take on Python versions
a good text editor or Python IDE ^[1]
A Google account
Operating systems:
- Linux
- Mac OS X
- Windows
Python 2.7, or 3.4 or higher

Summary

I searched around for a while for a good Python Google Calendar module. I did find several that showed some potential but most seemed a bit dated. I ended up on Google's own developer page ^[2]. On this page I found a Python example doing much of the process I was looking for. I thought to myself:

"Self, there is no reason to recreate something good. Let's take what is good here and modify it to fit our needs."

-- Me

So that's what I did.

I modified the code I found on that page to export a CSV^[3] file with the values I need. You may also noticed that I adjusted it to nicely pull in more than the 10 events the sample script did. You'll see this later.

I am mostly satisfied with the end result. It does what it needs to much faster and arguably better than I could manually--and that's the point.

Much of the instruction on this page comes straight from the Google article.

Ready, Go!

Step 1 - Setup Environment

The first thing to do is install the Google Calendar Python client library.

Before doing that, however, you need to setup a virtual environment. I always recommend using a virtualenvironment when installing Python libraries, especially when you're not sure what it does :) .

If you haven't setup a virtual environment to work on, please do so now. Visit virtualenvironment if you aren't sure how to do this. Go ahead and do this now.

I'll wait...

Photo by Jaime Top / Unsplash

Make sure you are in your virtualenvironment before doing the next steps.

In a terminal window, enter the following command:

$ pip install --upgrade google-api-python-client dateutil

You should not be prompted for your password to install the package. If you are, then you either didn't setup the virtual environment or you are not working in it. Go back to that page and make sure it is setup correctly.

Step 2 - Turn on the Google Calendar API

Follow the "Turn on the Google Calendar API" steps at https://developers.google.com/google-apps/calendar/quickstart/python.

Step 4 - Create the Python file `pull_from_gcal.py`

Create the Python file to run this script. You may want to create a new project if you are using PyCharm.

Step 5 - Save the code

Here you can copy/paste the finished code if you want to jump straight to running it. However, you may benefit from a line-by-line (mostly) walk-through to make sure you know what it is doing.

Step 6 - Run the script

Run the by right-clicking on the file in PyCharm and select Run 'pull_from_gcal.py'.

If all goes well, the script will open a browser window where you will be prompted to approve the application. Go ahead and do this. The script will gather the necessary credentials and save them to a local file for future use.

You browser should show a screen like the following if it worked.

And the CSV with yesterday's events will be created as well.

Wrap-up, Conclusion, Follow-up

Well, there you go. You now have a script that will save you time if you ever need to do this. Thank you for taking the time to read through and possibly testing this script.

Here are some potential enhancements to the script you may want to consider making.

Pull events from all of your calendars
Add field with list of attendees
Add field indicating if the event is a recurring event
Add field indicating if you created the event or if you were invited

I am working on an ebook with lots of scripting recipes. I plan to include this one with some of these changes. Stay tuned to learn more. I encourage you to subscribe using the form below.

I Love Pycharm for learning Python. You may see me using vim a lot because I have so much muscle memory with it. PyCharm is so much better for testing and editing code with greate syntax highlighting and code hints. ↩︎
https://developers.google.com/google-apps/calendar/quickstart/python ↩︎
find more articles on manipulating CSV files at ↩︎

My Take on Python Version 2 vs 3

Kajigga — Sat, 23 Sep 2017 20:44:52 +0000

It's a huge point of conversation.

The internet has been debating the merits of Python 2 vs Python 3 for years now. I am sure the Python foundation would love for everyone to have switched entirely to Python 3 by now. There are reasons to migrate to it and, at least some feel, reasons to stay on 2.7.

I'll summarize the differences then give my own opinionated take on the whole thing. (Hint: I think everyone should be using Python 3)

Python 2.7

Python 2.7.13, the most recent version of the 2.x series, came out way back in 2010.

Problems with Python 2.7

No new features have been introduced since that time. The Foundation releases bug fixes but new functionality will only be added to Python 3.

Python 3.x

Pros

Iterators, generators
- Many lists that would have been in memory lists with 2.7 are generators in Python 3. This make high memory functions much more efficient. There are some disadvantages but I won't go into detail here.
True UTF8 support - all strings are utf8 by default (Huge win!)
New features will be added to Python 3
It is not 7 years since the last release

Cons

Some libraries have not yet been migrated over to support Python 3. This is a huge problem for some people that have code written to rely on these libraries. However, it is my opinion that if the library isn't popular enough to demand a Python 3 version then it may not be necessary. That or get with the game people.
- I should note that it has been years since I ran into a library that I wanted to use that didn't work in Python 3.

My Opinion

Use Python 3 unless you absolutely have to use Python 2 for some reason. Python 3 is faster and better supported. All of the solutions I share on Scripting With Python are written with Python 3 first in mind. I may occasionally test them with Python 2 but I won't guarantee it.

5 Tasks That would be fun To Script with Python

Kajigga — Tue, 19 Sep 2017 19:20:32 +0000

This is so clichÃ© sounding but the sky really is the limit when it comes to scripting tasks. The feeling of watching the computer do some boring time-consuming task for you is awesome. Sit back and watch it go.

Listed below are 10 fun tasks that are fun to script and/or automate. I will eventually write detailed posts on each one.

Watch a Webpage for changes

Have you ever wanted to know when a new product was released, when a price changes, or when the most recent episode of Parks and Recreation is released? Why go through the effort of doing that yourself? Write a script to download the webpage of your choosing and watch for some value to change, appear, dissapear, whatever. When it changes, take some action.

Download a random image and set as desktop background image

Find a website (think Flickr, imgur, Google Photos, etc) that hosts images. Write a script to randomly download one and set it as your desktop background image.

Download latest News from Website, send via Email

Many services will send you notifications about their most recent updates. You could create your own script that downloads these and send them to you via email.

Create CSV Listing all of your photos

Photo by Smart / Unsplash

Get a list of your photos in Flickr, Google Photos, etc <fill in service name here>. If they have an API, great. If not, you could scrape the site.

Make mass changes to a CSV or Excel file

There are all sorts of things you can do with data saved as a Comma-Separated Values (CSV) or Excel file. A CSV file is just text but it can be tricky to edit. An Excel document is often a bit more than just text but sometimes it isn't. You could open it in a spreadsheet program like Libre Office, OpenOffice, Numbers, Google Sheets, or Microsoft Excel and make the changes, gulp, manually but that is not what were doing here. Automate that task! Do something better with your time.

There will be lots of articles on ScriptingWithPython.com that deal with CSV or Excel files as the basis of some scripting or automation task.

Have fun

The moral of the story is there are endless ways to script and automated tasks. Have fun! Be productive.

I recommend subscribing to my newsletter. You will receive updates when new posts are made.

DEV Community: Kajigga

Using a Proper Text Editor

Recommended Text Editors (2017)

I have a Editor Confession to Make

Easy Exporting CSV Data with Python

What is CSV?

Enough Review Already, Get to the Writing Data Good Stuff

Do not try to write CSV without a library.

The csv module is Awesome

Enough Enough already... Get to An Example

Reading CSV Data with Python

What is CSV?

Rules for reading CSV Files

Using the builtin Python csv module

Step 1: Open a csv file for reading

Step 2: Pass the CSV file handler to the Reader

Step 3: Iterate over each row

Create a CSV of Yesterday's Google Calendar Events

What is the Task?

Solution to pulling Google Calendar Events to CSV

Pre-requisites

Summary

Ready, Go!

Step 1 - Setup Environment

Step 2 - Turn on the Google Calendar API

Step 4 - Create the Python file pull_from_gcal.py

Step 5 - Save the code

Step 6 - Run the script

Wrap-up, Conclusion, Follow-up

My Take on Python Version 2 vs 3

Python 2.7

Problems with Python 2.7

Python 3.x

Pros

Cons

My Opinion

5 Tasks That would be fun To Script with Python

Watch a Webpage for changes

Download a random image and set as desktop background image

Download latest News from Website, send via Email

Create CSV Listing all of your photos

Make mass changes to a CSV or Excel file

Have fun

The `csv` module is Awesome

Using the builtin Python `csv` module

Step 4 - Create the Python file `pull_from_gcal.py`