DEV Community: Sayem Hoque

Setup Django Application Form to send emails to your Gmail Inbox

Sayem Hoque — Mon, 16 Sep 2024 02:28:39 +0000

For September 2024 none of the guides I found online were up to date and comprehensive for this so just writing the steps to make it work here:

https://myaccount.google.com/
Under "Recent Security Activity" make sure 2FA is on
Go to the search bar above, and search "App Passwords"

You'll be prompted as such - "To create a new app specific password, type a name for it below..."

Name it something like "my-django-site-form"

It will generate a password you can use in your application. Store this securely somewhere. This can be used in your Django settings, but I opt for saving as an .env variable.

To test to make sure this setup is working, you can run the following code in the Django shell:

python manage.py shell

from django.core.mail import send_mail
send_mail(
    'Test Email',
    'This is a test email.',
    'sender@email.com',
    ['recipient@example.com'],
    fail_silently=False,
)

Replace the two emails, and you should see an email sent to recipient@example.com

Python One-Liners #1: Create dictionary from csv file

Sayem Hoque — Wed, 15 Feb 2023 19:30:17 +0000

Turn two columns of any csv file into a dictionary:

col1_col2_map = pd.read_csv("cur.csv", dtype=str)[["col1", "col2"]].set_index("col1").to_dict()["col2"]

End to end - host a static site using firebase and a Google domain

Sayem Hoque — Thu, 09 Feb 2023 01:46:30 +0000

There are a million ways to host a static website. One of my favorites is via firebase hosting, because the spark plan is free and lets you experiment a good amount before you have to pay stacks. I've hosted personal sites, my development agency landing page https://www.flexbone.dev, and my reading highlighter extension page https://www.readwme.app via this method.

The only accounts you need are a Firebase account, and a domain you purchased via Google domains.

https://firebase.google.com/

https://domains.google.com/

Now that you have those two, create a new Firebase project with the + Add project widget.

To setup your static site, use any static html website you want to host and put it in a directory on your machine. cd into that directory and enter the following commands.

npm install -g firebase-tools
firebase login

Log into your firebase account in the web and your terminal should give you a success message.

firebase init

The firebase cli will have you select the project you created in the firebase console above. Choose the project you created before.

Now your directory should have a firebase.json as well as a public directory. Copy your index.html or your entire static site files into this directory.

Deploy your website.

firebase deploy

Your website should be live at something.firebaseapp.com

Redirect your website to your custom domain

This part always annoys me because it's poorly documented and different hosting domains call the records different names etc. So for firebase hosting and google domains, hopefully this explanation works.

Open up the console and look at the left side of the screen, you should see a drop down called Build, which has a link to the Hosting section.

You'll see a dashboard like this below that has the domains associated with your project. Before you add your custom domains, you won't have the two at the bottom here.

Add both www.yourgoogledomain.com and yourgoogledomain.com as custom domains in this section of the dashboard. The great part here is if the domain was purchased from google, it should skip through the verify ownership part of the domain and confirm it is yours.

When you add a custom domain, you'll see this above. You need to enter this data on the domains site. Go to domains.google.com and go to your domain you added to your firebase project.

Click DNS
Manage custom records

Create a first record where the values are as follows:
Host name - empty
Type of record - A
TTL - 3600 (default is fine)
Data - The IP address from firebase we saw above. Isn't it annoying that firebase called it "Value" and domains calls it "Data". Enter it here.

Create a second record that is the exact same as the above, except with the Host name filled in as www.

Host name - www
Type of record - A
TTL - 3600 (default is fine)
Data - Same IP address as above

Now, when you go back into the dashboard on firebase, your custom domain should have a pending status. You should be able to navigate to your custom domain and see your site right now, but until the pending goes away, you won't have https:// access, so you'll see an unsecure version of your site. Give it an hour and it should be fine. That's all

Dot products, cosine similarity, text vectors

Sayem Hoque — Thu, 20 Oct 2022 22:39:51 +0000

Cosine similarity is a measure between two single dimensional vectors that gives us a value ranging 0-1 to inform of the similarity between the vectors. The formula is below:

Cosine Similarity = (A . B) / (||A||.||B||)

Where (A . B) is the dot product between vector A and B. A dot product is the sum of the element-by-element product between A and B. For example,

A = [1, 2, 3]
B = [4, 5, 6]


A . B
>> 32
# (1 * 4) + (2 * 5) + (3 * 6) = 32

Meanwhile, ||A|| is the notation used to denote the L2 Norm of a vector. The L2 norm is a method to calculate the length of a vector in Euclidean space. Think of this as the length of a vector of length N as a "line" if the vector was drawn out on a N-dimensional graph. You sum the squares of the values in each dimension, and take the square root of the sum.

A = [1, 2, 3]

norm(A)

>> 3.7416573
# (1^2 + 2^2 + 3^2)^0.5 = 3.7416573

Numpy has a bunch of helpers so we don't need to run all of these calculations manually:

import numpy as np
from numpy.linalg import norm

# define two lists or array
A = np.array([1,2,3,4])
B = np.array([1,2,3,5])

# cosine similarity
cosine = np.dot(A, B) / (norm(A) * norm(B))
print("cosine similarity:", cosine)

>> 0.9939990885479664

A cosine similarity score near 1 means the vectors are very close to one another if they were projected. 0 means they are very dissimilar.

Descriptive stats + percentiles in numpy and scipy.stats

Sayem Hoque — Thu, 13 Oct 2022 04:55:51 +0000

To get the measures of central tendency in a pandas df, we can use the built in functions to calculate mean, median, mode:

import pandas as pd
import numpy as np


# Load the data
df = pd.read_csv("data.csv")

df.mean()
df.median()
df.mode()

To measure dispersion, we can use built-in functions to calculate std. deviation, variance, interquartile range, and skewness.

A low std. deviation means the data tends to be closer bunched around the mean, and vice versa if the std. deviation is high. The iqr is the difference between the 75th and 25th percentile. To calculate this, scipy.stats is used. Skew refers to how symmetric a distribution is about its' mean. A perfectly symmetric distribution would have equivalent mean, median, and mode.

from scipy.stats import iqr

df.std()
iqr(df['column1'])
df.skew()

from scipy import stats

stats.percentileofscore([1, 2, 3, 4], 3)
>> 75.0

The result of the percentileofscore function is the percentage of values within a distribution that are equal to or below the target. In this case, [1, 2, 3] are <= to 3, so 3/4 are below.

numpy.percentile is actually not the inverse of stats.percentileofscore. numpy.percentile takes in a parameter q to return the q-th percentile in an array of elements. The function sorts the original array of elements, and computes the difference between the max and minimum element. Once that range is calculated, the percentile is computed by finding the nearest two neighbors q/100 away from the minimum. A list of input functions can be used to control the numerical method applied to interpolate the two nearest neighbors. The default method is linear interpolation, taking the average of the nearest two neighbors.

Example:

arr = [0,1,2,3,4,5,6,7,8,9,10]
print("50th percentile of arr : ",
       np.percentile(arr, 50))
print("25th percentile of arr : ",
       np.percentile(arr, 25))
print("75th percentile of arr : ",
       np.percentile(arr, 75))

>>> 50th percentile of arr :  5
>>> 25th percentile of arr :  2.5
>>> 75th percentile of arr :  7.5

Now, using scipy.stats, we can compute the percentile at which a particular value is within a distribution of values. In this example, we are trying to see the percentile score for cur within the non-null values in the column ep_30.

non_nan = features[~features['ep_30'].isnull()]['ep_30']
cur = features['ep_30'][-1]

print(f'''Cur is at the {round(stats.percentileofscore(non_nan, cur, kind='mean'), 2)}th percentile of the distribution.''')

>>> This is at the 7.27th percentile of the distribution.

Transactions in Postgres

Sayem Hoque — Tue, 30 Aug 2022 23:01:52 +0000

In databases, a transaction is a single unit of logic -- they can potentially consist of multiple operations. An example is transferring a balance from a bank account to another. The transaction needs an amount removed from the first account and then added to the other. You wouldn't want an amount removed from the account and a program failing and the amount not ever adding to the second account. This is referred to as atomicity in databases.

In Postgres, implicit to every single statement is the idea that the statement occurs in a transaction. If we write two simple UPDATE statements, they might look something like this:

UPDATE table1 SET column1 = 100.00
    WHERE id = 3;

UPDATE table1 SET column1 = 100.00
    WHERE id = 4;

However, when any statement such as these UPDATE statements occur, you can think of it happening within a transaction like this:

BEGIN;
UPDATE table1 SET column1 = 100.00
    WHERE id = 3;
COMMIT;

BEGIN;
UPDATE table1 SET column1 = 100.00
    WHERE id = 4;
COMMIT;

A transaction "block" is the set of queries executed between the BEGIN and COMMIT commands. When you don't specify either of these commands, they implicitly surround every command.

We can override this default behavior to specify different transaction blocks using the BEGIN and COMMIT commands. The idea is that we can perform multiple operations on a database all at once. This is useful for times where if any intermediate operations might fail, we don't want the subsequent commands to execute. The entire "transaction" will not occur.

Below, you can see here we actually specified a BEGIN and COMMIT in this script. As a transaction block now, these two will be "committed" to the database and visible at the same time.

BEGIN;
    UPDATE table1 SET column1 = 100.00
        WHERE id = 3;

    UPDATE table1 SET column1 = 50.00
        WHERE id = 4;
COMMIT;

This can also be useful if an API or other service happens to request data in between the first and second UPDATE statement, and we don't want data to be accessed until all of the data is updated. Obviously the latency is really slow for simple UPDATE statements, but for a longer complicated process, this becomes increasingly important.

Another example: if a table is dropped but you don't want the table unviewable until it's repopulated completely:

BEGIN;
    CREATE TABLE table2 (
       column1 datatype(length) column_contraint,
       column2 datatype(length) column_contraint,
    );
    // add some data into table2

    DROP table table1;

    SELECT * INTO table1 FROM table2;

COMMIT;

Rollbacks

To roll back or undo the change of the last transaction block, you can use the ROLLBACK command. This can be extremely useful because it might be really difficult to traverse individual statements and figure out how you modified your tables. Wrapping up everything in a transaction block and then rolling back makes it easy to just undo everything that was in the block.

How to use GNU screen to run processes

Sayem Hoque — Fri, 26 Aug 2022 23:33:00 +0000

Screen is an ancient but still useful software on the GNU operating system that's available on most linux distros and on Macs.

For example, if you need to SSH to a remote server and need to run a long process that and can't worry about whether the connection will terminate, running the process in a screen will run your process in the background such that even if a connection is lost, it'll keep operating. When you reconnect to the screen, the state of the session will be as-is when you exited.

Steps

screen

sh@SH-MacBook-Pro tutorial % screen

you should see a start message here, which you can just press enter to get past.

You're now 'attached' to the screen session. To see that you're attached, you can see a list of screens and their statuses by running screen -ls:

sh@SH-MacBook-Pro tutorial % screen -ls
There are screens on:
        69475.ttys010.SH-MacBook-Pro      (Attached)
1 Socket in /var/folders/vw/865hm_fd2pj87qtwf_5n329m0000gn/T/.screen.

While you're attached, you can run a script and have it start processing whatever it is that you're doing.

sh@SH-MacBook-Pro tutorial % python somescript.py
             >> This is some logging being produced by
             >> a script that you're running

As your process is producing output, instead of hitting Ctrl+c, you can hit Ctrl + a + d to "detach" from the screen. The process will keep running in your still-attached screen session. You'll now be back in your normal terminal, and see a log that says you just detached from a screen:

[detached]
sh@SH-MacBook-Pro tutorial %

If you run screen -ls again here, you'll see that there is an active screen that you're detached from.

sh@SH-MacBook-Pro tutorial % screen -ls
There is a screen on:
    68979.ttys002.SH-MacBook-Pro    (Detached)
1 Socket in /var/folders/vw/865hm_fd2pj87qtwf_5n329m0000gn/T/.screen.

To reconnect to your screen, you can run screen -r. If you only have one active screen, you can just run screen -r. If you have multiple active screens to choose from, run screen -r with the screen id (68979 in this case).

sh@SH-MacBook-Pro tutorial % screen -r 68979

This will take you to your screen session that you created before.

sh@SH-MacBook-Pro tutorial % python somescript.py
             >> This is some logging being produced by
             >> a script that you're running
             >> More stuff was logged since we last checked

To exit a screen, Ctrl + d will exit the session. This kills the current screen session, and you won't be able to re-attach to it.

Scrolling

Within a screen session, scrolling up the terminal window with the mouse will not work as usual.

Scroll up a screen you're attached to by entering "scrollback" mode.

Ctrl + a + Escape will enter you into scrollback mode. Once you're in here, arrow keys will allow you to actually go up the screen log history. Mouse still won't work sorry.