DEV Community: Rajiv Abraham

Personal Experience: Software Engineer Job Search in Canada during COVID-19: Apr 9th - June 1st(53 days)

Rajiv Abraham — Sun, 31 May 2020 19:26:43 +0000

This article was originally posted at https://blog.rajivabraham.com/posts/job-search-2020

Purpose.

This article is about sharing my job search experience in case it's interesting or useful to anyone.

About me

I had decided to come back to a full time job after more than a year at my startup 15Rock. I have been developing professionally for 12 years now. Though twelve years is a big number, opportunities and circumstances, made me a generalist instead of a specialist i.e. I had maximum two years experience in any technology that I had worked on. So, I never fully fit the Senior/Architect Java/Ruby/Python profiles where companies expected in depth knowledge of technologies.

Also, in my free time, I like exploring concepts(Functional Programming, Logic Programming) which I believe are significant levers of productivity and developing my own libraries(Mercylog, Thampi, Jaya) when I could have spent that time, perhaps, learning current industry standards like Apache Spark, Deep Learning, Tensorflow or explored SQL in depth.

Job Search Strategy

I was applying for Machine Learning and Data Engineer roles as that was my recent experience and I'm really excited to apply all the software engineering processes that I obsess about(Yes, TDD ;) ) to the field of machine learning and data engineering. I had around 6 months of machine learning engineering knowledge and 2.5 years in data engineering in Python.

Hiring Environment

I was having lunch with an ex co-worker who was working in an interesting company, I wanted to apply to. When I broached the topic, he said, "Just to let you know, I got laid off last week". Companies had hiring freezes and were laying off people as well. This led to increased competition for the few companies that were actually hiring. My friend in Shopify, mentioned that they were seeing 10-20 times the normal volume of job applications. I did panic a bit and applied for 1-2 jobs though I didn't necessarily feel drawn to them.

Data

Since LinkedIn, where I post too, has a minimal blogging interface, I have my raw data in this Google Sheet and will use a textual format below.

Job Search Period: Apr 9th - June 1st(53 days)
Over all, I had applied to about 22-25 companies and 11 responded. In the Google Sheet above, you can see the first response in days. Most companies did not send a response after the first automated reply.
I got one offer from Bear Health Technologies

Companies

Here I'll try to list my experiences for select companies.

BenchSci

BenchSci is a startup in the pre-clinical R&D space. They were very quick to move with the interview process which I liked. I had 4 technical rounds. I liked talking to all of them. They were smart, friendly and it felt like a high quality team. If you are interviewing with them, they like to give you simpler variants of problems they see in their day to day work. Prepare yourself to get work assignments in an email and quickly import them to your laptop. So, before the interview, prepare:

Be quickly able to create a Python project, setup virtual environment and run unit tests(pytest).
Setup an existing postgres database and query editor(e.g. Postico for Mac). You may be given a CSV file so prepare in advance on how to import CSV files into a database.

Shopify

Shopify, according to Glassdoor, had a hiring surge during COVID-19. What I liked best about the interview process was the detailed feedback I got which I really valued. Also, since I had another offer in hand while I was interviewing with Shopify, the coordination team worked hard to accommodate my request to expedite the process. I'm thankful as I know they were getting a lot of applicants. All the interviewers were very friendly as well.

Amazon

They have a pre-screening programming interview to the normal pre-screening recruiter call :).
During the online pre-screening programming interview, I mistakenly pressed Escape and then the back button and it logged me out. There is a two minute wait period before you try again :). It's a small thing but I thought you should know.

Zapier

Zapier has been a remote first company even before the pandemic. I found it on a niche job site which promoted companies with better work life balance. I suspended the process after I got an offer but I really liked my conversation with the recruiter(Stephanie McDonald). She mentioned a phrase which really stuck: "Small Ego, Big Empathy". She used it to describe the kind of people they were looking for.

Finally..

I accepted an offer as a Machine Learning Engineer with Bear Health Technologies, a startup making a digital operations software that automatically indexes and summarizes large amounts of medical information reducing the time and cost to prepare and assess a medical brief. Subsequently, I had to suspend my interview process with Keebo, RelationalAI and Zapier.

Lessons(and Advice?)

Since the middle of May, I have noticed more activity as companies seem to be ready to open up hiring gradually in the next months. Hopefully these experiences below will still be of use to you.

Public Announcement on LinkedIn: I publicly announced on a LinkedIn post that I'm looking for a role. This prompted my ex manager at theScore to reach out to me and ask if I was interested in working again at theScore. Though theScore had a hiring freeze at the time, it was reassuring to know that maybe in a few months time, something could work out with a team I loved and enjoyed working with. This made me more choosy with the roles I was applying to. Another contact connected me to someone who helped me out with other companies and recruiters. A third contact reached out to me to apply to his own startup. Another contact offered to refer me to his company.
LinkedIn: If there was a company I liked, I would check if there was anyone in my network related to that company. Apart from direct contacts, I got my interview call for Ecobee because I knew someone who knew someone at Ecobee i.e 2nd degree contact. So don't hesitate to reach out to your network to help you to connect with others.
Building a Network: Out of the 11 companies that responded, 5 were referrals and 2 reached out from my network. Though I'm uncomfortable with the word 'networking', I love connecting :). I realized very early in my career that I must make friends outside my immediate team within the company and outside. Though people think I'm extroverted, I find it uncomfortable to go to new meetups especially when I'm tired in the evening after work. But I kept at it, often, telling myself to meet one new person and then I could go back home. Just one new person, that's it. And this habit has helped me immensely. When I moved to Toronto, I made new connections, found mentors who reignited my drive to go deeper into programming and found all my jobs in Toronto through meetup connections. I even met my co-founder! My job offer at Bear Health came from a contact I have met often at meetups for the past eight years. It takes time to do this but it's never too late. Go for meetups(also virtual) just because they excite you and it's inevitable you'll make friends. We in the tech community are very lucky that there are SO MANY of them.
Side Projects: As software programmers, we have the wonderful opportunity to try out and work in technologies that we want to get into in the future. I was interested in Julia, did one project on GitHub. RelationalAI was working with Julia and my Julia project stood out when they checked my Github info. Please do consider doing that side project in the technology you want a job in and beef up your resume.
Update LinkedIn: Amazon reached out because of my LinkedIn Profile.
Blog: Bear Health Technologies liked my profile because they read my blog. Blogging also ignited my desire to try out LinkedIn for my technical posts. This is an experiment as LinkedIn has minimal support for blogging(IMHO) and programmers generally don't blog on LinkedIn as it heavily seen as a venue for recruiters/job postings etc. But I realized that my entire network is here and 99% of them will never come to my blog :). I see 10-20 views per post but right now I'm still fine with that. If I can pass some of my knowledge to others, though few, I'm happy. Why don't you try it out too and let me know how it goes?
Cold Calls: No one likes to be cold called. What about warm calls? I sent my resume via LinkedIn to a few people who I had met once or twice. I was totally prepared for them to come back and say that they don't know me enough and don't feel comfortable to send my resume across or some white lie. What I felt bad a bit about was no replies at all. I guess it's hard to say no to people :). And I may one day be in the same position. So if someone can let us know, how to politely decline referral requests, that would be a valuable conversation.
Unit Tests: Write Unit Tests in interview questions. They seem to help immensely.
Salary Expectations: Since I was out of the job market for a while due to my startup, I just straight up said that I don't know what to ask and requested the recruiter to tell me what the salary ranges are. External recruiters gave me a good idea about the market while an internal recruiter mentioned their salary range for that role. In this way, I knew where to place myself.
On Hiring Freezes: I realized that hiring freezes were not absolute. One of my friends approached his manager at a company which had a hiring freeze and the hiring manager said that he would consider making an exception for the right candidate :). I wasn't that candidate but it's good to know that there is some wiggle room.
Life Project: Consider having a side project which excites you. I have a side project(Mercylog) which I call my life project :). I would work on it after an interview if I thought it didn't go well. When I started waking up later and later as I wasn't working, I decided on working on my side project first thing in the morning or not at all. That made me wake up early :).
On Connecting: Every interview is an opportunity to meet someone for an hour. Even for jobs that I didn't get, I connected with any interviewer that I liked. In fact, one interviewer, who I thought wasn't impressed with me, invited me to go out for coffee after the pandemic. This just shows me that interviews come and go but I should still take the opportunity to connect with people I like even if their company didn't consider me qualified at that point of time.
Misc: RelationalAI reached out to me when I posted a few questions on a sleepy google group(datalog-cafe)

Resources/Practice

These are the resources I used during my interview prep:
Interview Cake: I was apprehensive about coding interviews and finishing in time. I decided to embrace the process and get into the minds of experienced programmers. I chose to buy Interview Cake as it was on a steep discount for the pandemic. I liked it as it gave explanations and gave the thought process. I have heard of Cracking the Coding Interview as well. If you don't want to pay, I'll recommend trying out Codewars, LeetCode, HackerRank etc. They are community driven interview question banks with points and skill levels. I have only used Codewars and it was nice.
For the System Design Interview, I recommend this amazing resource: https://github.com/donnemartin/system-design-primer
For Data Engineers, this looked nice: https://github.com/andkret/Cookbook
Salary Negotiation. I found this video by Josh Doody very informative.
Since most companies used CoderPad, I created my own free account and practiced in the free sandbox. That's how I discovered that CoderPad has pytest support and I just kept on using it.

Gratitude
Coursera: I'm very grateful to Coursera for accepting my application to study the Deep Learning course by Andrew Ng for free.
Coding Cake: I appreciate the drastic reduction in their prices and personalized feedback when I replied to one of their emails.

In Conclusion

I hope that my experiences help you in some way. Please know that you are not alone in feeling scared, anxious during these times. We all feel that. The last thing to do is to close yourself to friends and family. If this is your first lay off, I know it's tough and a major downer. But I know, from my experiences and seeing others, we always get jobs sooner or later. Sometimes, we get better jobs :). Best of Luck.

DataScript: A modern datastore for the browser

Rajiv Abraham — Tue, 12 May 2020 20:48:47 +0000

This article was originally posted at https://blog.rajivabraham.com/posts/datascript-python

Purpose.

This blog article will:

give a brief introduction to a new way of managing state on the browser by leveraging a library called DataScript.
Though DataScript is a ClojureScript/JavaScript library, we will learn how to call it using Brython, a Python implementation that runs on the browser.
sneak in a tutorial on mercylog-datascript which is a query composer library for DataScript.

DataScript(and Datalog)

How do you manage application state in the browser? I'm new to frontend development so I'm sure there are many solutions out there which I don't know about. But let me ask the question from a different perspective. How would a backend developer manage state? She uses a database. She writes SQL queries. Then for the browser, can we use an in-memory datastore and write queries on it?

DataScript is one such in-memory datastore for the browser. One can create it during page load, insert, delete rows etc. and when the user closes the page, it's gone. It's very light and fast. Also, instead of SQL, it offers a query style called Datalog which is a declarative, logic based query language. I got interested in Datalog when I saw that applications written in its variants reduced the code size by 50% or more(Overlog, Yedalog). This is why I got interested in DataScript in the first place. I have written a brief introduction to the concepts here. In short, It's like SQL + Recursion.

But Datalog is not the only reason to use DataScript. That would be like flying in the Boeing 747 for its in-flight entertainment. It has so many other cool features but that's for another post. This post will focus on just the Datalog query interface for DataScript.

Just to complicate things a bit, DataScript is actually a ClojureScript library which has a JavaScript interface too. So in all respects, you would use it cleanly in those languages. But, I'm interested in Python in the browser and therefore all the examples below will be in a Python implementation in the browser called Brython. I have written a brief introduction before:

All the examples below are from the following github repo. NOTE: All the files mentioned below take some time to load the data.

There is no spoon, Neo.

Cough, before we move on, DataScript is different from your conventional SQL datastores. In conventional SQL databases, when we wish to implement an entity e.g customer, we first create a table schema customer with the attributes: name and age for e.g.

CREATE TABLE customer (
    name varchar,
    age int
);

According to Datomic, the backend database which DataScript is an in-memory implementation of, such an approach is rigid. Hey, I sense it but not well enough to defend it. So, let's not have twitter wars on that one... yet.

DataScript eschews storing data as separate entities in separate tables(like customer or person). Instead, one can see DataScript as a store for attributes and its values only. So, DataScript prescribes that we only specify the schema for the attributes itself. E.g. name is string, age is integer but do not group them at all at design time before hand. Instead, the application decides which attributes to group together for a particular instance of that entity.

In an extreme example to clarify the concepts, the application may decide to store just a customer's name(Rajiv) and his eye color(black) and give the customer an id 1. For another customer, it may store her as id 3 and just store her age(40). Simplistically(and naively incorrect), you can see the database as a single god table with columns entity_id, attribute, value where each row can belong to a different (entity_id, attribute) composite key. E.g.

entity_id	attribute	value
1	name	Rajiv
3	age	40
2	name	Canada
1	eye_color	black

What about the entity 2? He/She is named Canada? Must be a cutie unlike the dull Rajiv. But we don't know, the application knows. It may not represent a person at all! In this example, the pair (entity id=2, attribute=name) could be one attribute and value for an instance of the entity country e.g. Canada(O Canada, our home and native land... True Patriot Love .. with your public healthcare you command!). Before you panic and ditch Datomic/DataScript, in practice, the attribute names contain the entity name as well. So it'll be ":customer/name" instead of "name" for Rajiv and ":country/name" instead of "name" for Canada. Or you could still leverage a common name attribute, your choice. And there is better support for ids than the pitiful example I'm giving here. Also, Imposter Alert, I'm not an expert by any means. I'm learning by doing :). That's my EULA. Please do check out the design principles behind Datomic/DataScript. I think Datomic is a masterpiece in database engineering.

Let's show you how to add data. For e.g., we can add two dictionaries(Igor and Ivan) to the database.

db = datascript.empty_db()
db1 = datascript.db_with(db, [{":db/id": 1,
                                "name": "Ivan",
                                "age": 17},
                                {":db/id": 2,
                                "name": "Igor",
                                "age": 35}])

You'll notice that datascript takes a database (db) and adds data to it and returns a new database db1. This is because databases in DataScript are immutable. Once created, you can't change it. This makes debugging and reasoning about the code using a database easier. I've written a bit about the general concept of immutability here

For each dictionary, DataScript will add two rows(one for name and another for age) in the single god table. The way DataScript keeps track is with the entity id i.e. :db/id attribute name which is a reserved attribute name in DataScript.

So this may be represented in the god table like:

entity_id	attribute	value
1	name	Ivan
1	age	17
2	name	Igor
2	age	35

Finally, the query. The query follows the Clojure style of a mixed list. If we want to know the age of the entity whose name is Igor,
the query would look like [:find ?a :where [?e "name" "Igor"] [?e "age" ?a]]. The variables with a question mark are called logic variables. If you are familiar with inner joins in SQL, it's very similar. We are saying that if there is some :db/id (i.e. ?e) whose name is Igor, for that value of ?e(i.e 2) find a relation for age (i.e. [?e "age" ?a]) and return his age(i.e ?a). We should get back 35. The query and the call to the database is:

result = datascript.q('[:find ?a :where [?e "name" "Igor"] [?e "age" ?a]]', db1)

The full example is below:

<!doctype html>
<html>

<head>
    <meta charset="utf-8">
    <script type="text/javascript" src="brython.js"></script>
    <script type="text/javascript" src="brython_stdlib.js"></script>
    <script src="https://github.com/tonsky/datascript/releases/download/0.18.10/datascript-0.18.10.min.js"></script>
</head>

<body onload="brython(1)">
<script type="text/python">
from browser import window, alert
datascript = window.datascript

db = datascript.empty_db()
db1 = datascript.db_with(db, [{":db/id": 1,
                                "name": "Ivan",
                                "age": 17},
                                {":db/id": 2,
                                "name": "Igor",
                                "age": 35}])
result = datascript.q('[:find ?a :where [?e "name" "Igor"] [?e "age" ?a]]', db1)
alert(result)
</script>
</body>

</html>

Other notes for the above example:

We load the Brython libraries: brython.js and brython_stdlib.js to run Python in the browser.
We load the datascript library from GitHub using the script tag . Brython will automatically create a reference for datascript under the window object. So we can refer to the module using window.datascript
When you open igor-just-datascript.html in a browser, it should return [[35]]. The return value is always a list.

Mercylog Datascript

Now you could happily continue and query the database with strings like [:find ?a :where [?e "name" "Igor"] [?e "age" ?a]]. Personally, I find writing code as strings works for simple queries. But it's not easily composable and reusable. So I wrote a Brython library called mercylog-datascript which allows us to use Python to construct the queries. So the same query in mercylog-datascript becomes

# The original datascript style as str_query
str_query = '[:find ?a :where [?e "name" "Igor"] [?e "age" ?a]]'

# mercylog-datascript style
from mercylog_datascript import DataScriptV1
m = DataScriptV1()
A, E = m.variables('a', 'e')
query = m.query(find=[A], where=[[E, "name", "Igor"], [E, "age", A]])
assert str_query == query.code()

Granted, it's more code than a simple string but I think when the queries become complex and begin to have reusable parts, this style may start paying off.

All you need to access the mercylog-datascript library is add the following script tag:

<script src="https://github.com/RAbraham/mercylog-datascript/releases/download/v0.1.4/mercylog_datascript.brython.js"></script>`

The full script is at igor-mercylog-datascript.html

Mercylog-DataScript Queries

Now that you have seen an example, let me show you the current query feature set with a sample dataset of actors, directors and movies. The full dataset is here. Here is an elided subset.

[
 {":db/id":  100,
  ":person/name": "James Cameron",
  ":person/born": "1954-08-16"},

 {":db/id":  131,
  ":person/name": "Charles Napier",
  ":person/born": "1936-04-12",
  ":person/death": "2011-10-05"},
  ....

  {":db/id":  200,
  ":movie/title": "The Terminator",
  ":movie/year":  1984,
  ":movie/director":  100,
  ":movie/cast":  [101,
               102,
               103],
  ":movie/sequel":  [207]},

 {":db/id":  201,
  ":movie/title": "First Blood",
  ":movie/year":  1982,
  ":movie/director":  104,
  ":movie/cast":  [105,
               106,
               107],
  ":movie/sequel":  [209]},
....

]

Here is the code to access the raw dataset and a simple pattern access as above with the Igor example. The code is here. Below we want to know the birthdate of Linda Hamilton and it will return [['1956-09-26']]

<!doctype html>
<html>

<head>
    <meta charset="utf-8">
    <script type="text/javascript" src="brython.js"></script>
    <script type="text/javascript" src="brython_stdlib.js"></script>
    <script src="https://github.com/RAbraham/mercylog-datascript/releases/download/v0.1.4/mercylog_datascript.brython.js"></script>

    <script src="https://github.com/tonsky/datascript/releases/download/0.18.10/datascript-0.18.10.min.js"></script>
</head>

<body onload="brython(1)">
<script type="text/python">
from browser import window, alert, console
datascript = window.datascript

from mercylog_datascript import DataScriptV1
m = DataScriptV1()
import urllib.request, json
data_file_url = 'https://raw.githubusercontent.com/RAbraham/mercylog-datascript-client/master/data.json'
console.log('Loading File')
with urllib.request.urlopen(data_file_url) as url:
    result = json.loads(url.read())
console.log('End loading file')

db = datascript.empty_db()
db2 = datascript.db_with(db, result)

e, name, born = m.variables('e', 'name', 'born')

query = m.query(find=[born], where=[[e, ":person/name", "Linda Hamilton"], [e, ":person/born", born]])
q = query.code()
result = datascript.q(q, db2)
alert(result)  # [['1956-09-26']]


</script>
</body>

</html>

Henceforth, I'll only focus on the mercylog-datascript query builder and how it supports DataScript.

Let's start with something simple. How do we just get the id(i.e :db/id) of a person? As shown in entity.html, the following query returns [[102]] for Linda Hamilton.

e = m.variables('e')
query = m.query(find=[e], where=[[e, ":person/name", "Linda Hamilton"]])

That's great, but sometimes you want to create a parameterized query i.e. a query that can be used with different values. Let's generalize the query above and then we can use it for different values. We do this by adding a parameters key to our query function.

In parameterized_queries.html, you'll see:

# [:find ?e :in $ ?name :where [?e ":person/name" ?name]]
e, name = m.variables('e', 'name')
query = m.query(find=[e], parameters=[name], where=[[e, ":person/name", name]])
q = query.code()
result1 = datascript.q(q, db2, 'Linda Hamilton')
result2 = datascript.q(q, db2, 'Sylvester Stallone')

As you can see above, the same query can be used to query Linda Hamilton and Sylvester Stallone.

Sometimes, you don't care about the entity, when doing a search. For e.g, you just want all the movie titles. In that case, mercylog-datascript provides the m._ variable.

In underscore.html, you'll find:

# [:find ?title :where [_ ":movie/title" ?title]]
title = m.variables('title')
query = m.query(find=[title], where=[[m._, ":movie/title", title]])

This would return

[['First Blood'], ['Terminator 2: Judgment Day'], ....  ['Terminator 3: Rise of the Machines']]

In attr.html, we show how to find out the attributes which are commonly associated with a particular attribute(i.e. ":person/name"). To reiterate, as DataScript does not have a fixed schema, it's possible for a customer record c1 to only have a name but another record c2 to have a name and age.

# DataScript Query: [:find ?attribute :where [?person ":person/name"] [?person ?attribute]]
person, attribute = m.variables('person', 'attribute')
query = m.query(find=[attribute],
                where=[[person, ":person/name"],
                       [person, attribute]])

q = query.code()
result = datascript.q(q, db2) # [[':person/born'], [':person/name'], [':person/death']]

Above, we find out that an entity which has an attribute :person/name can also have one or more of [[':person/born'], [':person/name'], [':person/death']]. The slow reader who didn't try to read this in the elevator may have noticed that previously our list in the where clause were of size three but when we do such kind of meta searches, we can just pass lists of size 2(e.g. [person, attribute])

Now, you sigh and say, this is all good but no language is a language unless it allows you to create functions. In transformation.html, we see an example of passing a user defined function(get_age) to be executed against the database.
In the code snippet below, notice [get_age(born), age]. You could see this as age = get_age(born) for different values of born and you could reuse that in your query. In our case, we just ask for it directly in find.

Since it's user defined, we add get_age to the parameters argument for query as well and pass get_age.function to the DataScript query engine as well.

# DataScript Query:[:find ?age :in $ ?get_age ?name :where [?p ":person/name" ?name] [?p ":person/born" ?born] [(?get_age ?born) ?age]]
from datetime import datetime
current_year = int(datetime.today().strftime('%Y'))
get_age = m.function('get_age', lambda born: (current_year - int(born.split('-')[0])) )

born, p, age, name = m.variables('born', 'p', 'age', 'name')
query = m.query(find=[name, age],
                parameters=[get_age, name],
                where=[[p, ":person/name", name],
                       [p, ":person/born", born],
                       [get_age(born), age]])

q = query.code()
result = datascript.q(q, db2, get_age.function, "Richard Crenna")
alert(result) #[['Richard Crenna', 94]]

Ok, you say this is simple stuff, yawn, what about SQL like aggregate functions? DataScript has you covered. In aggregates.html, we show the use of inbuilt functions of DataScript i.e. agg.max(age). As a bonus, we now show how to combine that with a user defined function(get_age).

# DataScript Query:[:find (max ?age) :in $ ?get_age :where [?p ":person/name" ?name] [?p ":person/born" ?born] [(?get_age ?born) ?age]]
from datetime import datetime
current_year = int(datetime.today().strftime('%Y'))
get_age = m.function('get_age', lambda born: (current_year - int(born.split('-')[0])) )
agg = m.agg
born, p, age, name = m.variables('born', 'p', 'age', 'name')
query = m.query(find=[agg.max(age)],
                parameters=[get_age],
                where=[[p, ":person/name", name],
                       [p, ":person/born", born],
                       [get_age(born), age]])

q = query.code()
result = datascript.q(q, db2, get_age.function)
alert(result) # [[94]]

What about SQL where like clauses? We have filters in DataScript too. It's basically a user defined function in it's own row. For e.g, in predicate2.html(because getting predicate1.html to work made me cry), I'm going to make a simple filter function called born_before_1950 to filter out the wiser actors.

# DataScript Query: [:find ?person :in $ ?born_before_1950 :where [?person ":person/born" ?birth_date] [(?born_before_1950 ?birth_date)]]
person, birth_date = m.variables('person', 'birth_date')
born_func = lambda x: int(x.split('-')[0]) < 1950
born_before_1950 = m.function('born_before_1950', born_func)
query = m.query(find=[person],
                parameters=[born_before_1950],
                where=[[person, ":person/born", birth_date],
                       [born_before_1950(birth_date)]])
q = query.code()
result = datascript.q(q, db2, born_before_1950.function)
alert(result) # [[148], [119], [146], [105], [136], [116], [135], [145], [111], [114], [118], [115], [147], [104], [139], [110], [133], [131], [140], [101], [123], [142], [137], [138], [106], [107], [113], [124], [130]]

We can leverage ClojureScript inbuilt functions too. As seen in inbuilt_function3.html(why 3?, you guessed it ;)), we can call inbuilt functions like <. You still leverage the m.function method but since you don't pass your own user defined function, mercylog-datascript will assume that you are trying to call an inbuilt function. You can call that convenient or over smart, only time will tell.

# DataScript Query: [:find ?title :where [?movie ":movie/title" ?title] [?movie ":movie/year" ?year] [(< ?year 1984)]]
title, movie, year = m.variables('title', 'movie', 'year')
lt = m.function("<")
query = m.query(find=[title],
                where=[[movie, ":movie/title", title],
                       [movie, ":movie/year", year],
                       [lt(year, 1984)]])

q = query.code()
alert(q)
result = datascript.q(q, db2)
alert(result) # [['First Blood'], ['Alien'], ['Mad Max'], ['Mad Max 2']

But most of the fun happens when we link different sources of data together to make money. For e.g. you may call a service which gives you the box office numbers for popular movies and you want to link that with the data in your database and find out the corresponding directors.

This is similar to an inner join in SQL between two tables but in this case, one side is a 'table' in DataScript and the other side is your in-memory structure which you obtain after calling the API for the service. Suppose you store the result of the API call in an in-memory list(title_box_office_pairs below). You need to tell DataScript about it's structure. So you specify it in the parameters argument as m.collection([title, box_office]) and then later pass title_box_office_pairs to datascript.q() later. Then you could use the logic variable title and use that to link to the data in the DataScript store. Code snippet below and full code here,

# DataScript query: [:find ?director ?box_office :in $ [[?title ?box_office]] :where [?p ":person/name" ?director] [?movie ":movie/director" ?p] [?movie ":movie/title" ?title]]

movie, p, title, box_office, director = m.variables('movie', 'p', 'title', 'box_office', 'director')
# title_box_office_pairs below could have been obtained from some api call.
title_box_office_pairs = [
 ["Die Hard", 140700000],
 ["Alien", 104931801],
 ["Lethal Weapon", 120207127],
 ["Commando", 57491000],
]
query = m.query(find=[director,
                      box_office],
                parameters=[m.collection([title, box_office])],
                where=[[p, ":person/name", director],
                       [movie, ":movie/director", p],
                       [movie, ":movie/title", title]])

q = query.code()
result = datascript.q(q, db2, title_box_office_pairs)
alert(result) # [['Richard Donner', 120207127], ['Mark L. Lester', 57491000], ['John McTiernan', 140700000], ['Ridley Scott', 104931801]]

Finally, we can define rules. Let's say we want to make a rule: Two people are mates if their names match(hey, that's a good reason to be mates, no?). In Datalog, one would write it like

Mate(E1, E2) <= Name(N, E1), Name(N, E2)

i.e. any person E1 is a mate of any person E2 if they both have the name N. Again N here is a logic variable which could represent all the names in the database.
In mercylog-datascript, you would write the above as:

e1, e2, n = m.variables('e1', 'e2', 'n')

mate = m.relation('mate')
r = m.rule(mate(e1, e2), [[e1, "name", n],
                          [e2, "name", n]])

A complete code listing is below. We also show to add a m.function to choose those rows where e1 has a bigger id than e2. I know! It does not make sense but I've been labouring on this post for three weeks and it's time to wrap up :P:

NOTE: There is some boilerplate code for now to send the rule as a parameter to the query:rule_code = '[' + r.code() + ']'. I'll look into improving it if a thousand of you star this project on Github(Yeah, ain't going to happen, I know :))

from mercylog_datascript import DataScriptV1
m = DataScriptV1()
db = datascript.empty_db()

db2 = datascript.db_with(datascript.empty_db({"age": {":db/index": True}}),
                 [{ ":db/id": 1, "name": "Ivan", "age": 15 },
                  { ":db/id": 2, "name": "Petr", "age": 37 },
                  { ":db/id": 3, "name": "Ivan", "age": 37 }]);

e1, e2, p, title, n = m.variables('e1', 'e2', 'p', 'title', 'n')
mate = m.relation('mate')
gt = m.function('<')
query = m.query(find=[e1, e2],
                where=[mate(e1, e2),
                       [gt(e1, e2)]])
q = query.code()
alert(q)

r = m.rule(mate(e1, e2), [[e1, "name", n],
                          [e2, "name", n]])
rule_code = '[' + r.code() + ']'
alert(rule_code)
result = datascript.q(q, db2, rule_code)

alert(result) # [[1,3]]

That's about it. If you came so far, I love you. I really do. Call me.

If you loved the above.

DataScript has two more APIs in addition to the Datalog API:
- Entity API: I think this is similar in concept to graph database like query languages. I leave it to the reader to investigate
- Pull API: Similar to GraphQL
If you want to learn about more about the DataScript syntax, check out this site
The Datomic Data Model which DataScript is inspired about is mentioned here

Brython: On replacing JavaScript with Python for front-end development

Rajiv Abraham — Tue, 05 May 2020 17:13:35 +0000

This article was originally posted at https://blog.rajivabraham.com/posts/brython

Purpose.

This blog article will:

give a brief introduction to using Brython, a Python implementation for front-end development on the browser

The entire project is here

Introduction

Jealous of JavaScript programmers, a cabal of Python programmers secretly met to discuss the future of Python in this apocalyptic world. JavaScript was everywhere and eating Python's lunch. With Node.js, JavaScript had invaded Python territory and ended its dominance as everyone's favorite language after Ruby(not very dominant then, is it? Mr. Author). It was time to make a thrust into the heart of JavaScript land: The Browser.

Don't forget your history(and future)

The cabal were not the only gentlemen concerned with this dilemma. The author of Transcrypt believed in poison and espionage. He decided to write a Python compiler which compiled to JavaScript code. Like good poison, there was no trace of Python. It looked very promising. But the author of this boring post found a bug and raised an issue which was unsolved/unattended for ages. So he felt it wise to abandon this side plot.

Others wanted to learn from history. Just immigrate the entire family. At least, that's what Pyodide thought of doing. Their strategy was to create an enclave on the side with a full Python Interpreter which can run Python code. Thus you could run any Python code including most of the data science stack which contains C language bindings (e.g. Numpy, Pandas).

This looks very promising too. But on initial lazy tests by this author, the initial page load was a bit slow(The real reason was that the author also could not find an easy way to make this work and was happy to not pursue it any further.)

So the cabal decided to do what every cabal is supposed to do i.e. create another Python to JavaScript compiler but this time, compile it to JavaScript when the page loads(unlike Transcrypt which compiles to JavaScript ahead of time). Thus, the fellowship of Brython was formed. One snake to rule them all.

Hello World

Let's code up the customary 'Hello World'

The Brython paratroopers(compiler) is here.

<script type="text/javascript"
       src="https://cdn.jsdelivr.net/npm/brython@3.8.9/brython.min.js">
</script>

We activate it on page load

<body onload="brython()">
...
</body>

Within the body tag above, we write the Brython Code:

<script type="text/python">
from browser import document

document <= "Hello World"
</script>

We just add Hello World to the document element. Hmmm. That was easy.

In complete form, it's shown below.

<!doctype html>
<html>
<head>
    <meta charset="utf-8">
    <script type="text/javascript"
        src="https://cdn.jsdelivr.net/npm/brython@3.8.8/brython.min.js">
    </script>
</head>

<body onload="brython()">

<script type="text/python">
from browser import document

document <= "Hello World"
</script>
</body>
</html>

This will simply print "Hello World" to the page.

Calculator

Let's now make a calculator(code courtesy: Brython). The full code is here

Yes, you were right. We do need a table. Let's make one.

from browser import document, html
calc = html.TABLE()

Let's add the first row only. Just the display box(we'll name it result) and C.

calc <= html.TR(html.TH(html.DIV("0", id="result"), colspan=3) +
                html.TD("C"))

Yes, I'm not very sure of this <= syntax either. But hey, for such a lovely library, I'll settle for it too :).

Let's now add the number pad

lines = ["789/", "456*", "123-", "0.=+"]
calc <= (html.TR(html.TD(x) for x in line) for line in lines)

Finally, we add calc to the document

document <= calc

Now that's all good. How do we make it work? First, we need to capture a reference to the result element to manipulate it when the number pad is pressed.

result = document["result"] # direct access to an element by its id

Next, we need to update the result whenever any element in the number pad is clicked. Let's make an event handler. We'll trust the Brython developers that this code works. Notice the manipulation of result based on the button you clicked.

def action(event):
    """Handles the "click" event on a button of the calculator."""
    # The element the user clicked on is the attribute "target" of the
    # event object
    element = event.target
    # The text printed on the button is the element's "text" attribute
    value = element.text
    if value not in "=C":
        # update the result zone
        if result.text in ["0", "error"]:
            result.text = value
        else:
            result.text = result.text + value
    elif value == "C":
        # reset
        result.text = "0"
    elif value == "=":
        # execute the formula in result zone
        try:
            result.text = eval(result.text)
        except:
            result.text = "error"

Finally, we associate the event handler above to the click event of all buttons.

for button in document.select("td"):
    button.bind("click", action)

See, how easy it is when someone else writes the code :P. But seriously, Brython is a wonderful work of engineering and perhaps the best display of programmer love for their beloved Python language. Please support the developers, at least with a star on their Github repo!

For the advanced reader

One can also integrate third party libraries like Vue.js as shown here.
A great in depth explanation of the concepts can be found here

Introduction to PureScript: Twitter Search API

Rajiv Abraham — Wed, 01 Apr 2020 17:55:43 +0000

This post is an ported, edited version of the original

TLDR: I wrote this in a fiction format for fun. The actual code is in the repo. Also, I'm new to FP so this is newbie code. I can refactor it to be elegant but I want to keep this simple for beginners.

Twitter Storm

Kim Kardashian felt uneasy as soon as she woke up. She had just used the r-word yesterday and suffered a huge backlash. She felt vulnerable about her twitter following and needed to be reassured. She had to do something different. Yes, she could just type her name in the Twitter App and see what people were saying about her. But she had secretaries for that. No, she had to do what no other celeb had done before. She would code!

What language though? A language which is nice and clean and pure. So she googles around and discovers PureScript! She installs it in a breeze while wondering about this Mr. Java Script guy who was always complaining online on how difficult it was. Sigh. Ok, what next?

Reading Twitter credentials

First, she has to read her Twitter credentials from a file. Yes, she could hard code the passwords in the program but she's a celeb. She knows Security.

So, she got her credentials from Twitter and created a file like below at config/twitter_credentials.json

{
  "consumer_key": "KimMama",
  "consumer_secret": "KimLikesToCode",
  "access_token": "KimDoesNotKnowWhatThisIsFor",
  "access_token_secret": "KimThinksTwitterHasGoneMad"
}

She built a JavaScript like object in PureScript(called records) using type:

type TwitterCredentials =
  { consumer_key :: String
  , consumer_secret :: String
  , access_token :: String
  , access_token_secret :: String
  }

How do we read this file?

import Node.Encoding (Encoding(..))
import Node.FS.Aff (readTextFile)

readConfigStr :: String -> Aff String
readConfigStr path =  readTextFile UTF8 path

import Node.Encoding (Encoding(..)) meant import the type constructor Encoding and the .. meant import all it's data constructors as well, one of which is UTF8. Since she is a celeb and she is never wrong, type constructors are like abstract base types and data constructors are like normal OOP constructors but fancier. You can have data constructors with different names and you can even treat them like Enumerations in switch/case like statements(Kim's BFF liked to call them pattern matching).

Aff stands Asynchronous Effect(the synchronous effect is called Effect). These effects represent an action that the program would like to take, but not executed yet. Whaaa?

If Kim wanted to call Khloe for lunch, buy flowers for her mother and type her next tweet... She wouldn't be the person doing it, would she? It would be her secretary! All, she would do is text her secretary commands to do this thing but it wouldn't happen until her secretary actually executed the commands at a later time!

In the same way, Aff(and Effect) were like texts by Kim to her new secretary PureScript. It was a way of telling PureScript that she wanted them to be done but it was just a representation of a command, not the actual execution of the command. By representation, it just meant it was a value, just like the way number 3 or "a_string" or a JavaScript object were values.

For e.g., imagine the following pseudocode in an imperative language(e.g. Python):

1: x = print("A String")
2: x
3: x

The output would be

A string

The execution and evaluation of the print statement both happen at line 1.

But in a functional language, the above pseudocode would be something like

1: run(
2: let x = print("A String")
3: x
4: x
5: )

And the output would be

A String
A String

let is like the variable assignment in imperative code.

Only the evaluation happens at lines 2-4 but not the execution. The execution happens inside run. So before the program is given to run, x replaced at lines 3 and 4 to be print("A String"). Note, the print has different interpretations. In the imperative setting, it executes a command, but in the functional setting, it executes nothing, just returns back a value representing an action for future execution by the run procedure.

Another viewpoint is that most applications always start with the main function. In PureScript, perhaps the simplest program one could write is.

import Effect.Console (log)
main:: Effect Unit
main = log "Product Placement Here. ;)"

The signature for log is log :: String -> Effect Unit. Unit stands for nothing, as in, we don't expect anything back from the console.

And like the pseudocode above, what happens within PureScript code, unseen by the programmer is something like

run(main)

Kim felt a chill through her spine. She regretted not taking programming seriously in school.

Ok, readConfigStr returned a Aff String but she needed to convert it to our TwitterCredentials record. She asked her secretary for technology to find a library for her and she found PureScript-Simple-JSON by a guy called Justin Woo.

import Simple.JSON as SimpleJSON
import Data.Either (Either(..))

parseConfig :: String -> Either String TwitterCredentials
parseConfig s =
  case SimpleJSON.readJSON s of
    Left error -> Left (show error)
    Right (creds :: TwitterCredentials) -> Right creds

parseConfig has an Either String TwitterCredentials in it's signature. It's like an union type. The result could either be a String(an error string) or the actual credentials. PureScript defines Either as

data Either a b = Left a | Right b

So if we want to return a string, we return Left "my error string", the actual credentials as Right creds. That way, the person calling parseConfig knows which is which.

In parseConfig, SimpleJSON.readJSON returned an Either but Kim didn't want to deal with the complex Left type, so she just converted that to a string using show.

Now it was just a matter of calling readConfigStr and passing the value to parseConfig. Something like this pseudocode

cStr = readConfigStr path
return parseConfig cStr

But she couldn't make it compile! She started panicking and thought of what would happen if the word got out and Taylor Swift found out. The Shame

"Try the do notation", said a voice from behind.

Kim swivelled back and her mouth opened with surprise.

"Kanye! I didn't know you knew PureScript!"

"Nah, PureScript is for hipsters. I'm old school. I like my Haskell."

He continued, "The do notation allows you to extract the String from Aff String and gives you the illusion of the pseudocode above."

readConfig :: String -> Aff (Either String TwitterCredentials)
readConfig path = do
  cStr <- readConfigStr path
  pure $ parseConfig cStr

"What's pure $ for?", asked Kim?

Kanye sighed. He knew the author of this post was in a hurry to move on to doing cooler stuff and didn't want to get into monads in this post. So he bailed too.

First $. That's just a simple way of saying consider everything after as one value. For e.g.
show $ SimpleJSON.readJSON s meant show (SimpleJSON.readJSON s) instead of (show SimpleJSON.readJSON) s. Kim approved. She liked $ signs.

Kanye then braced himself for his 'simplification' of pure.

"You noticed that it was cStr <- readConfigStr path and not let cStr = readConfigStr path. The <- is syntax sugar which make it look like an =. But what is really happening underneath is something very similar to callbacks. The Aff String type has to be given a function to work on the String value within it. But this function can't just be cStr -> parseConfig cStr. The function has to return back an Aff something. pure is a constructor. In this context of Aff, when we say pure something, it's like saying new Aff(something) or in our case, it's like saying new Aff(parseConfig(cStr))"

Kim beamed at Kanye. He looked so hot right now. She wanted him so bad.

Bearer Token from Twitter.

Great, that gave her the credentials but she needed a bearer token from Twitter which she would then use to get the results. How does one call the Twitter endpoint in PureScript? She beckoned her secretary for technology to find her a library. Her secretary came back running.

"I found a library called Milkis... again by Justin Woo!"

Kim's eyes sharpened with intent. She wondered out aloud, "Do you think this Justin guy is a celebrity in the PureScript world? Hmmmm make my agent call his agent. Let's do a reality show together."

Kim first created a method to construct the authorization string from the credentials and encode it in Base64. The <> was like an append operator.

import Data.String.Base64 as S
authorizationStr :: TwitterCredentials -> String
authorizationStr credentials =
  S.encode $ credentials.consumer_key <> ":" <> credentials.consumer_secret

She then made a simple fetch helper method from Milkis.

import Milkis as M
import Milkis.Impl.Node (nodeFetch)


fetch :: M.Fetch
fetch = M.fetch nodeFetch

She then created a method to get the bearer token string or return a string as error(in the Left part of the code).

import Milkis as M
import Effect.Aff (Aff, attempt)

getTokenCredentialsStr :: String -> Aff (Either String String)
getTokenCredentialsStr basicAuthorizationStr = do
    let
      opts =
        { body: "grant_type=client_credentials"
        , method: M.postMethod
        , headers: M.makeHeaders { "Authorization": basicAuthorizationStr
                                 , "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8"
                                 }

        }
    _response <- attempt $ fetch (M.URL "https://api.twitter.com/oauth2/token") opts
    case _response of
      Left e -> do
        pure (Left $ show e)
      Right response -> do
        theText <- M.text response
        pure (Right theText)

Now to bring it all together.


type BearerAuthorization =
  { token_type :: String
  , access_token :: String
  }

basicHeader :: String -> String
basicHeader base64EncodedStr = "Basic " <> base64EncodedStr

toBearerAuthorization :: String -> Either String BearerAuthorization
toBearerAuthorization tokenString = do
  case SimpleJSON.readJSON tokenString of
    Left e -> do
      Left $ show e
    Right (result :: BearerAuthorization) -> do
      Right result

getTokenCredentials :: TwitterCredentials -> Aff (Either String BearerAuthorization)
getTokenCredentials credentials = do
  tokenCredentialsStrE <- getTokenCredentialsStr $ basicHeader $ authorizationStr credentials
  case tokenCredentialsStrE of
    Left error -> do
      pure (Left error)
    Right tokenCredentialsStr -> do
      let tokenCredentialsE = toBearerAuthorization(tokenCredentialsStr)
      case tokenCredentialsE of
        Left error -> do
          pure (Left error)
        Right authResult -> do
          pure (Right authResult)

Great, we had the bearer token. It's finally time to search for Kim Kardashian!
PureScript had this interesting signature format though. What it was saying below was that showResults took as input a BearerAuthorization and a String and returned an Aff (Either String SearchResults)

Also, the SearchResults and Status had lots of fields but she just wanted the basic stuff.


type Status =
  { created_at :: String
  , id_str :: String
  , text :: String
  }

type SearchResults =
  { statuses :: Array Status
  }

twitterURL :: String -> M.URL
twitterURL singleSearchTerm = M.URL $ "https://api.twitter.com/1.1/search/tweets.json?q=" <> singleSearchTerm

showResults :: BearerAuthorization -> String -> Aff (Either String SearchResults)
showResults credentials singleSearchTerm = do
  let
    opts =
      { method: M.getMethod
      , headers: M.makeHeaders { "Authorization": "Bearer " <> credentials.access_token}

      }
  _response <- attempt $ fetch (twitterURL singleSearchTerm) opts
  case _response of
    Left e -> do
      pure (Left $ show e)
    Right response -> do
      stuff <- M.text response
      let aJson = SimpleJSON.readJSON stuff
      case  aJson of
        Left e -> do
          pure $ Left $ show e
        Right (result :: SearchResults) -> do
          pure (Right result)

Finally, reaching the very end to the main command!

import Effect.Class.Console (errorShow, log)
import Effect.Aff (Aff, launchAff_)

main :: Effect Unit
main = launchAff_ do
  let searchTerm = "Kim Kardashian"
  config <- readConfig "./config/twitter_credentials.json"
  case config of
    Left errorStr -> errorShow errorStr
    Right credentials -> do
      tokenCredentialsE <- getTokenCredentials credentials
      case tokenCredentialsE of
        Left error ->
          errorShow error
        Right tokenCredentials -> do
          resultsE <- showResults tokenCredentials searchTerm
          case resultsE of
            Left error ->
              errorShow error
            Right result ->
              log $ show $ "Response:" <> (show result.statuses)

launchAff_ was required because the entire computation returned Aff something but main was of type Effect Unit. So launchAff_ just converted Aff something to Effect Unit

As Kim beamed with pride at her code, she flashed her eyes at Kanye and asked him, "Isn't the code beautiful?"

Kanye gazed into her eyes and said, "Actually, it sucks. There are so many case statements in that code that I feel cross eyed."

And the next thing Kanye knew, was that he was flat on the ground, his jaw felt like it had been displaced and he was seeing double.

For there are three things you don't tell your wife:

1) Honey, you have gained weight
2) Your code sucks
3) I miss my mother's cooking.

As Kanye massaged his jaw, he muttered, ".. I guess she does not want to know about the ExceptT Monad.."

PureScript on AWS Lambda: Using Express and Serverless

Rajiv Abraham — Tue, 31 Mar 2020 15:15:30 +0000

This post is an ported, edited version of the original

Purpose

This post shows you how to run PureScript 0.12 on AWS Lambda, using Express on the Serverless platform.

Setup

The final output of this article is also a repo

Prerequisites.

You should have nodejs (i.e. nvm, npm) setup on your machine, preferably 8.10. AWS Lambda uses 8.10. I’m new to nodejs land but I’m guessing that higher minor versions should be ok.

Software Setup

npm install -g purescript
npm install -g pulp bower
npm install -g serverless

AWS Credentials

If you haven’t already, generate an AWS key and secret. This user must have AdministratorAccess permissions. Here are the docs or a slightly outdated video. For the video, follow on to 1:40 and ignore the Serverless Dashboard instructions(around 0:58 to 1:00), we are going to do that on the command line.

serverless config credentials --provider aws --key your-aws-key --secret your-aws-secret

Project Setup

mkdir hello-purescript-serverless
cd hello-purescript-serverless
npm init # fill in as directed
pulp init
pulp build

If all goes well, you should see something like below:

* Building project in /Users/rabraham/Documents/dev/purescript/hello-purescript-serverless
Compiling Data.Symbol
Compiling Type.Data.RowList
...
Compiling Main
Compiling PSCI.Support
Compiling Effect.Class.Console
* Build successful.

Now let’s install our project specific packages.

bower install --save purescript-aws-lambda-express purescript-express
npm install --save aws-serverless-express express
npm install serverless-offline --save-dev

purescript-express is a wrapper on express while purescript-aws-lambda-express provides the wrapper for AWS Lambda. serverless-offline allows us to test the code locally before deploying it to AWS Lambda.

At time of writing, purescript-express has an issue where we have to install the following two packages too. Try a pulp build right now and if that fails, run the following commands

bower install --save purescript-test-unit
bower install --save purescript-aff

Let’s build it

pulp build

You should see some warnings but at the end, you should see * Build successful.

Main Course

In your src/Main.purs, delete the previous code and paste the following:

module Main where

import Node.Express.App (App, get)
import Node.Express.Handler (Handler)
import Node.Express.Response (sendJson)
import Network.AWS.Lambda.Express as Lambda

-- Define an Express web app

indexHandler :: Handler
indexHandler = do
  sendJson { status: "ok" }

app :: App
app = do
  get "/" indexHandler

-- Define the AWS Lambda handler

handler :: Lambda.HttpHandler
handler =
  Lambda.makeHandler app

Build Your App

pulp build

Serverless Setup

In the root of your project, create the file serverless.yml and paste the following:

service: purescript-aws-lambda-express-test

provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 128
  # stage: ${opt:stage dev}
  region: us-east-1

functions:
  lambda:
    handler: output/Main/index.handler
    events:
      - http:
          path: / # this matches the base path
          method: ANY
      - http:
          path: /{any+}
          method: ANY

plugins:
  - serverless-offline

Let’s test this locally: On one terminal.

serverless offline start

Open another terminal and do:

curl http://localhost:3000

You should see {"status":"ok"}

Deploy

Once it works locally, let’s deploy to AWS.

serverless deploy -v

Output should look like something below. Note, your endpoint will be different:

Serverless: Packaging service
...
Service Information
service: purescript-aws-lambda-express-test
stage: dev
region: us-east-1
...
Stack Outputs
...
ServiceEndpoint: https://l4qajv7v95.execute-api.us-east-1.amazonaws.com/dev
....

Copy the link shown as ServiceEndpoint and you know what to do!

curl https://l4qajv7v95.execute-api.us-east-1.amazonaws.com/dev

Output:

{"status":"ok"}%

Undeploy

serverless remove -v

I hope this enables you to make PureScript web applications! Thanks to purescript-express and purescript-aws-lambda-express for making this possible.

Reading code easily with immutable values(Pyrsistent).

Rajiv Abraham — Mon, 23 Mar 2020 01:59:52 +0000

TLDR:

Use data structures that don't change once created(using Pyrsistent) to make it easy to understand and maintain code.

Purpose.

When a data structure(e.g. dict) once created, does not change, it allows us to read code with more confidence.

For e.g. Let's say you have a customer variable in your code and you are tracking it's value by reading the code. What do you reason is the value of customer below?

customer = dict(name="Rajiv", age=40)
some_function(customer)
print(customer)

In the above code, we can't say. In Python, for most default data structures like dict, it is possible that some_function could have changed the value of customer. So, we have to dig in and read the code of some_function to be fully sure. If the code of some_function was below:

def some_function(a):
    a1 = a
    a1['name'] = 'NewRajiv' # Changing the values. blasphemy
    # do something with a1

then print(customer) would display {'name': 'NewRajiv', 'age': 40}.

If you are lucky, some_function does not pass it forward to other functions! Or else, you would have to dig in and read those functions too :). Now that would suck. Unless, it is the intention that the customer field should be mutated but in most cases, one does not expect it to be so(in other languages,naming conventions are used to indicate if that is the case). A knowledgeable programmer may make a copy(via the copy.deepcopy()) and work on the copy to prevent her code from affecting the client code but I have not been that knowledgeable programmer :)

What if we could use a data structure that once created, cannot be changed i.e. it is immutable. Let's check out a library called pyrsistent that gives us such data structures.

from pyrsistent import m # m is like a dictionary

customer1 = m(name='Rajiv', age=40)
customer2 = customer1.set(name='NewRajiv')
print(customer1) # pmap({'age': 40, 'name': 'Rajiv'})
print(customer2) # pmap({'age': 40, 'name': 'NewRajiv'})

When we specify a different value('NewRajiv'), a copy is created with that new value and assigned to customer2. customer1 still retains the value it was first assigned. Now, let's go back to our previous code example and modify it a bit for pyrsistent

from pyrsistent import m # m is like a dictionary

def some_function(a):
    a1 = a.set('name', 'NewRajiv')
    # do something with a1 

customer = m(name="Rajiv", age=40)
some_function(customer)
print(customer)

print(customer) would display {'name': 'Rajiv', 'age': 40}, the value set in our code. So, we can safely reason about our code and what it's doing without worrying about it changing inside some_function. We don't have to even look into some_function in this case. Trust me, when you can't run that snippet of code to see what the actual values are, this feature makes life so easy :).

pyrsistent also has support for other common data structures(i.e. lists, sets) and much much more. Most of these pyrsistent data structures are drop in replacements for their Python counterparts when it comes to accessing the data.

From the pyrsistent docs:

from pyrsistent import v  # like a list

a = v(1, 2, 3)
b = a.append(4)

print(b[1])  # 2
print(b[1:3])  # pvector([2, 3])
print([2 * x for x in b])  # [2, 4, 6, 8]

On Speed and Memory

I simplified(ok, I lied) when I said that pyrsistent makes a copy of the data structure. Such a practice would be a waste of memory and time if we copy over every huge data structure. pyrsistent mitigates that to a great extent by not just blindly copying data structures and then making the modifications. It tries to be intelligent by sharing the common parts between a original data structure and the new modified copy to save on memory and time.

Let's take an example(Credit: Wikipedia: Persistent Data Structures). Ah, you now are exposed to what this is really called. This concept is called persistent data structures or functional data structures.

NOTE: The below example is just to explain the concepts and such a binary search tree is not part of pyrsistent.

Let's say you had a binary search tree(xs) which was a persistent data structure:

Now if you added a node e to that data structure, e.g. ys = insertNode(xs, e) A naive implementation would copy the data structure and then insert e at the appropriate location. In a persistent data structure approach, it would be:

Since e falls into the right side of the tree(i.e. the tree with g as the root), the tree with root b is not affected and hence can be reused. You can see a arrow from d' to b indicating that.

This reuse saves memory space, and saves time not done copying as it merely uses pointers to refer to the unchanged data.

Note: just because a data structure is being reused does not mean modifications to one can affect another. They are immutable and hence cannot be modified. If you do modify, a new structure is created like above.

Note on Note: pyrsistent tries to be as fast as possible and has comparable speeds to the norm for most cases. The complexity of most operations are well described in their docs.

Nested Transformations

What if we have to update a nested value in a data structure while maintaining immutability. pyrsistent has a method transform for that.
How I would normally do it

import copy
m4 = dict(a=1, b=6, c=[1, 2])
# I want to update c[1] to 17
m4_new = copy.deepcopy(m4) 
m4_new['c'][1] = 17

From their docs,

from pyrsistent import m  # m is like a dictionary
from pyrsistent import v # m is like a list
m4 = m(a=5, b=6, c=v(1, 2))
m4_new = m4.transform(('c', 1), 17)
print(m4_new) # pmap({'a': 5, 'c': pvector([1, 17]), 'b': 6})

Updating dictionaries

One thing I do very often is merging dictionaries. For e.g., I may have to construct my configuration taking the the following sources with the earliest being the highest priority.

Environment variables
File configuration
Default configuration

How I would normally do it.

default_conf = dict(database_url='dev_url', user='postgres', port=5432)
# Imagine file_conf below was extracted from a file
file_conf = dict(user='test_user', port=5433)
# Imagine env_conf below was constructed from environment variables
environment_conf = dict(database_url='test_url')
final_conf = {**default_conf, **file_conf, **environment_conf}

print(final_conf) # {'database_url': 'test_url', 'user': 'test_user', 'port': 5433}

That's great for 99% of the cases I would think :). But for the sake of discussion, perhaps if you had HUGE dictionaries(e.g. merging all the data you scrapped illegally from some website ;) ), that would be some duplication of data in memory.
In pyrsistent:

from pyrsistent import m
default_conf = m(database_url='dev_url', user='postgres', port=5432)
# Imagine file_conf below was extracted from a file
file_conf = m(user='test_user', port=5433)
# Imagine env_conf below was constructed from environment variables
environment_conf = m(database_url='test_url')
final_conf = default_conf + file_conf + environment_conf

print(final_conf) # pmap({'database_url': 'test_url', 'user': 'test_user', 'port': 5433})

I hope this is enough to get you started in a better coding experience :). There are many other wonderful features in pyrsistent like having the above behaviour for records(PRrecord) and clases(PClass) and many more advanced features. I'll leave that for another post.

So head out to pyrsistent and check it out. And if you like it, don't forget to star! It's a wonderful piece of engineering whose authors that we should applaud and support.

Introduction to Datalog(Bashlog) in Python

Rajiv Abraham — Sun, 16 Feb 2020 06:27:18 +0000

TLDR:

Datalog is like SQL + Recursion. It's derivatives have reduced the code base by 50% or more.

Datalog

Today, I would like to explore a constrained language called Datalog. It's a constrained form of Prolog and may not be as expressive as C++ or Python. But it's derivatives have been known to reduce the numbers of lines of code down by 50% or more(Overlog, Yedalog).

Let's get started:

Datalog has a minimalist syntax which I love. Let's take an example.
Suppose our data is about fathers and sons, mothers and daughters. If we had an excel sheet, we would enter the data like:

Father  Son
Aks     Bob
Bob     Cad
Yan     Zer

and another excel sheet for mothers and daughters:

Mother  Daughter
Mary    Marla
Marla   Kay
Jane    Zanu

In Datalog, we express the same data as(together):

father('Aks', 'Bob')
father('Bob', 'Cad')
father('Yan', 'Zer')
mother('Mary', 'Marla')
mother('Marla', 'Kay')
mother('Jane', 'Zanu')

Here we are trying to say Aks is the father of 'Bob' and 'Bob' is the father of 'Cad'. The datum father('Aks', 'Bob') is called a fact i.e. it is true.

So Datalog can be used to express data. Not very interesting so far but a building block. These facts above can also be viewed as the existing state of the system, like we store state in files, or databases.

But that's not enough. What about code? For Datalog, code are specified as rules to be applied declaratively.

Let's say our program needs to find out who's a grandfather. We could write a rule like:
'A person X is the grandfather of Z if X is the father of Y and Y is the father of Z'. In Datalog, this rule is written as:

grandfather(X, Z) :- father(X,Y), father(Y, Z)

The LHS (i.e. grandfather) is known as the head and the RHS after the :- is known as the body

X, Y, Z are special variables called logic variables. They are different from regular variables. They are more used to represent a pattern or to link the head and the body.

To further understand logical variables, consider these two rules:

grandfather(X, Z) :- father(X,Y), father(Y, Z)
grandmother(X, Z) :- mother(X,Y), mother(Y, Z)

Here the X, Z and Y used in grandfather are completely different from the X, Y and Z in grandmother. In rules, the variables only make sense in that single rule. So we can reuse the same logic variables in different rules without worrying that they have some logical connection.

The next concept is queries. How do we feed input and get back some output.
Queries are similar to rules but without a head.

father(X, 'Aks'), mother(Z, 'Aks')

we mean, find the mother and father of 'Aks'
or

father(X, 'Raj'), mother(Z, 'Raj')

we mean, find the father and mother of 'Raj'

Suppose, we want to say find the mother and father of all the children in the database, we make the query

father(X, Y), mother(Z, Y)

Datalog will link Y for all mother and father facts and find the mothers and fathers for a child. It will not mix up fathers and mothers :)

Now, If you opened a datalog interpreter and fed the above and made the following queries, you would get the results shown after the # sign

father(X,_) # ['Aks', 'Bob', 'Yan']
father(_,X) # ['Bob', 'Cad', 'Zer']
father(X, Y) # [('Aks', 'Bob'), ('Bob', 'Cad'), ('Yan', 'Zer')
father(X, 'Zer'), father('Zer', Y) # [] as there are no facts that match the query
grandfather(X, Y) # [('Aks', 'Cad')]
grandfather(X,_) # ['Aks']

Here '_' is a special variable indicating that you don't care for the result.

I was always interested in the Datalog syntax and it's power. I kept delaying it until I met Bashlog. Because, the syntax of datalog is so simple, it makes it easy to write interpeters for different targets. What Bashlog did was take Datalog syntax and convert it to bash scripts! Because, it used awk(mawk actually), sed, grep, which are tuned for high performance on Unix like platforms, it was incredibly fast in parsing big text files, comparable with all the specialized databases out there. Just Bash Scripts. It blew my mind. So if you are interested in pure Datalog, check out Bashlog

With Bashlog, you can run any bash like command and read that using Bashlog. Imagine there was a file('~/data.tsv') with tab separated values of

Aks Bob
Bob Cad
Yan Zer

We could read that data like:

facts(F, S) :~ cat ~/data.tsv
father(X, Y) :- facts(X, Y)

And then we proceed the same manner like before. What's awesome is that you can run any Unix command(e.g. ls -l) as long as it returns an output of tab separated values.

But I wanted to use Datalog in my day to day programming. I wanted to see if I could use and leverage Datalog along with Python. Some benefits of Datalog in Python are:

Modularity. How do we abstract out patterns in our rules and facts.
Possible access to exisitng rich source of libraries.

So I built Mercylog in Python.

So let's translate the above rules to Mercylog syntax.

Installation

If you are using the Bashlog variant,

then you need Java 8 already installed

git clone https://github.com/RAbraham/mercylog_tutorial.git
cd mercylog_tutorial
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python tutorial.py

That should print

['Aks']
['Mary']

Read below on the explanation and make tweaks if you want and run python tutorial.py again.

Usage

import mercylog
m = mercylog.BashlogV1()


# father('Aks', 'Bob')
# father('Bob', 'Cad')
# father('Yan', 'Zer')
# mother('Mary', 'Marla')
# mother('Marla', 'Kay')
# mother('Jane', 'Zanu')

father = m.relation('father')
mother = m.relation('mother')
facts = [
    father('Aks', 'Bob'),
    father('Bob', 'Cad'),
    father('Yan', 'Zer'),
    mother('Mary', 'Marla'),
    mother('Marla', 'Kay'),
    mother('Jane', 'Zanu'),
]

# grandfather(X, Z) :- father(X,Y), father(Y, Z)
grandfather = m.relation('grandfather')
X, Y, Z = m.variables('X', 'Y', 'Z') 
grandfather(X, Z) <= [father(X, Y), father(Y, Z)]

While in Datalog, you don't have to explicitly state the variables and the relation, as it is baked in to the language, in our library in Python, we need to (e.g. X, Y, Z and father, grandfather)

Making a query in python has the following syntax

m.run(facts, rules, query)

A concrete example would be:

m.run(facts, rules, grandfather(X, Y)) # which gives [('Aks', 'Cad')]
m.run(facts, rules, father(X,_)) # ['Aks', 'Bob', 'Yan']
m.run(facts, rules,father(_,X)) # ['Bob', 'Cad', 'Zer']
m.run(facts, rules,father(X, Y)) # [('Aks', 'Bob'), ('Bob', 'Cad'), ('Yan', 'Zer')
m.run(facts, rules, granfather(X,_)) # ['Aks']

Creating this DSL in python gives us some unique benefits. For e.g if we had these two relations

paternal_grandfather = m.relation('paternal_grandfather')
maternal_grandmother = m.relation('maternal_grandmother')
father = m.relation('father')
mother = m.relation('mother')

X, Y, Z = m.variables('X', 'Y', 'Z')
rules = [
    paternal_grandfather(X, Z) <= [father(X, Y), father(Y, Z)],
    maternal_grandmother(X, Z) <= [mother(X, Y), mother(Y, Z)]
]

If you notice, the rule for paternal_grandfather and maternal_grandmother are very similar. I could perhaps encapsulate that into a function. I'll use the word transitive though I believe it is incorrect to use it.. but I don't know what to call this for now. Rewriting the above code:


def transitive(head, clause):
    X, Y, Z = m.variables('X', 'Y', 'Z')
    return head(X, Z) <= [clause(X, Y), clause(Y, Z)]

paternal_grandfather = m.relation('paternal_grandfather')
maternal_grandmother = m.relation('maternal_grandmother')
father = m.relation('father')
mother = m.relation('mother')


rules = [
    transitive(paternal_grandfather, father),
    transitive(maternal_grandmother, mother)
]

In this way, using Python, we have modularized a pattern using the transitive function.

Let's recap the benefits of Mercylog

Simple Syntax. All you need to know is facts and rules. Because of such simplicity, it is also easy to build compilers for it.
Expressive. Rules give a powerful mechanism
Declarative. Like SQL but more expressive. So we can optimize it's engines without affecting the code

I'll continue to update you with my future learnings!

DEV Community: Rajiv Abraham

Personal Experience: Software Engineer Job Search in Canada during COVID-19: Apr 9th - June 1st(53 days)

Purpose.

About me

Job Search Strategy

Hiring Environment

Data

Companies

BenchSci

Shopify

Amazon

Zapier

Finally..

Lessons(and Advice?)

Resources/Practice

Gratitude

In Conclusion

DataScript: A modern datastore for the browser

Purpose.

DataScript(and Datalog)

There is no spoon, Neo.

Mercylog Datascript

Mercylog-DataScript Queries

If you loved the above.

Brython: On replacing JavaScript with Python for front-end development

Purpose.

Introduction

Don't forget your history(and future)

Hello World

Calculator

For the advanced reader

Introduction to PureScript: Twitter Search API

Twitter Storm

Reading Twitter credentials

Bearer Token from Twitter.

PureScript on AWS Lambda: Using Express and Serverless

Purpose

Setup

Prerequisites.

Software Setup

AWS Credentials

Project Setup

Main Course

Serverless Setup

Deploy

Undeploy

Reading code easily with immutable values(Pyrsistent).

TLDR:

Purpose.

On Speed and Memory

Nested Transformations

Updating dictionaries

Introduction to Datalog(Bashlog) in Python

TLDR:

Datalog

Installation

Usage