DEV Community: Jacob E. Shore

Some Thoughts About Data Modeling In Jupyter Notebooks

Jacob E. Shore — Sat, 30 Jan 2021 20:57:51 +0000

My job recently sponsored my taking a Nanodegree course at Udacity for data engineering. So far, the projects seem to be done mostly by interacting with various databases (currently PostgreSQL and Cassandra) through python in Jupyter Notebooks set up on their platform. The format they set up the exercises is really rather repetitive an tedious. Each insert is the same thing, many times. I wonder if this is a common pattern among data people, or just an attempt to get you used to writing the SQL. (Although, SQL knowledge and python was a supposedly prerequisite for the course, so this is a bit confusing.)

I've taken to working out the SQL using DataGrip locally first, then implementing the queries through abstracting the queries to classes. It seems wrong to use an ORM when the course isn't (especially, since I know data people often hate ORMs 😀), so I have set up the classes myself. Here is the basic idea:

Make a meta class that puts all the similar insert classes into a list.
Make a new class for every typical insert.
Then abstract away the actual inserts so you can reuse the logic.

Here is the basic code:

# The metaclass
class MetaInsert(type):
    counter = 0
    instances = []

    def __iter__(self):
        return iter(getattr(self, "instances", []))

# This just creates the table
# Should probably take a string for the table name and 
# namedtuples for columns and types - 
# so it could be leveraged for all tables.
def create_cheese_table():
    try:
        cur.execute(f"CREATE TABLE IF NOT EXISTS cheese (id SERIAL, name varchar);")
    except psycopg2.Error as e:
        print(e)

# Example insert class
class CheeseType(metaclass=MetaInsert):
    def __init__(self, name):
        CheeseType.instances.appens(self)
        self._id = CheeseType._counter + 1
        self.name = name

    def insert_row(self):
        insert_str = f"INSERT INTO cheese (name) VALUE ('{self.name}');"
        try:
            cur.execute(insert_str)
            print(f'{self.name} inserted into the cheese table!')
        except psycopg2.Error as e:
            print(e)

create_cheese_table()

cheddar = CheeseType('cheddar')
mozzarella = CheeseType('mozzarella')
muenster = CheeseType('muenster')

[cheese.insert_row() from cheese in CheeseType]

# cheddar inserted into the cheese table!
# mozzarella inserted into the cheese table!
# muenster inserted into the cheese table!

This is not the most robust solution, but it works for its use case.

Please LMK what could be improved 😀 or what you would've done differently.

First Impressions of F#

Jacob E. Shore — Sun, 10 Jan 2021 00:00:00 +0000

One of my goes for 2021 is to learn C# and F# (I mostly work in Python and JavaScript). More details about New Years resolutions here.

I've started to get my feet wet in F#. I've discovered .NET interactive notebooks in VS Code for C# and F# - which makes playing around with them more fun. (I wonder if there is a way to embed them in markdown, but I digress.) Now that I've had an opportunity kick the tires a bit, here are some of my discoveries:

There are some things that strike me as a little odd, but I'm sure they were intentional decisions.

Unlike most languages, in F# = is used for both assignment and equality. So you can do:

let my_bool = 10 = 2
let this_is_ten = 2
printf "%b" my_bool // false
printf "%i" this_is_ten // 2

I suppose this is similar to SQL. Similarly, != is not used to show inequality but rather <>.

Also, in F# you don't have to declare the type. F# figures the type out for you. But apparently, this can happen even after you declare something. For example:

// This is a function that adds two things together
// but what kinda of things?
let add_stuff x y = x + y
// Here, the compiler decides that
// I'm concatenating strings with this function
let hello = add "Hello " "world!" //  "Hello world!"
// If I then try to add numbers I will get a type error
let my_sum = add 10 30 // Error: expected strings, got ints

In an alternate reality - if I reversed the order, this would happen:

let add_stuff x y = x + y
let my_sum = add 10 30 // 40
let hello = add "Hello " "world!" // Error: expected ints, got strings

So it seems like, even though the definition of the function is identical in both cases, it is further defined by how I first use it. I find this counter-intuitive and would rather just define my function from the get-go.

The Microsoft docs make a big deal out of not needing to define your types. I'm not sure why this is a feature. I'd rather be able to just look at a function and know how it behaves without seeing how it was used. Maybe there is something I'm missing here.

Of course, I don't have to let the compiler infer the type, I can be as explicit as you want to be. Here I'm telling my functions exactly what I want them to recieve.

let double_list (l: List<int>) = List.map(fun x -> x * 2) l
let add_one (l: List<int>) = List.map(fun x -> x + 1) l

// I can put them together like this:
let double_then_add = double_list >> add_one
// I can also do it like this:
let add_then_double = double_list << add_one
// Which is the same as this:
let also_add_then_double = add_one >> double_list

let new_list = double_then_add [1..10] // [3; 5; 7; 9; 11; 13; 15; 17; 19; 21]
let another_new_list = add_then_double [1..10] // [4; 6; 8; 10; 12; 14; 16; 18; 20; 22]
let a_third_new_list = also_add_then_double [1..10] // [4; 6; 8; 10; 12; 14; 16; 18; 20; 22]

The >> operator is called the composition operator.
Theres also a piping operator like this:

[1..100]
|> List.filter(fun x -> (x%2) = 0) // filter out all odd numbers
|> List.map(fun x -> x * 2) // Multiply them all by 2
|> printfn "Even doubles: %A"

(**
    Will return this:

        Even doubles:
        [4; 8; 12; 16; 20; 24; 28; 32; 36; 40; 44; 48; 52; 56; 60; 64; 68; 72; 76; 80;
        84; 88; 92; 96; 100; 104; 108; 112; 116; 120; 124; 128; 132; 136; 140; 144; 148;
        152; 156; 160; 164; 168; 172; 176; 180; 184; 188; 192; 196; 200]
*)

I'm not 100% yet on what the difference between piping and composing is.

If anyone else has insights on F# that they feel could be helpful - or would like to clarify things for me, it would be most appreciated. :)

Using localStorage API - Good idea? Bad idea? Thoughts?

Jacob E. Shore — Tue, 26 Nov 2019 19:05:02 +0000

I've been playing around w/ the idea of using the browser localStorage API instead of a backend/database (for lightweight apps ). Here's a basic todo app made with Vuejs and the localStorage API.

Todo with localStorage

Has anyone done this? Good idea? Bad idea? Thoughts?

demo page

Eyes Follow You

Jacob E. Shore — Mon, 18 Nov 2019 18:50:06 +0000

Yeah, there's a share button on CodePen, so why not?