DEV Community: Shaikhul Islam

Back to basics - Python

Shaikhul Islam — Wed, 03 Feb 2021 04:40:07 +0000

This is part 1 of back to basics series. Today's topic is python. Let's dive into code!

Numbers - int, float, complex

year = 2021
type(year) # int
pi = 3.14
type(pi) # 3.14

# conversion
int(3.5) # 3
float(3) # 3.0

# some math op
float_num = 5 / 2 # returns a float
floored_div = 5 // 2 # returns a int
squared = 2 ** 2
mod = 5 % 2

# get divisor and modulas
div, mod = divmod(5, 2)

complex_num = 3 + 2j # (3 + 2j)
type(complex_num) # complex

# some functions
abs(-25)
round(2.5)

Strings and Bytes

# string can use single/double or docblock quote
name = "Python" + 'Python' + """Python"""
type(name) # str
binary = b"Python"
type(binary) # bytes
name == binary # false

# codepoint to str
chr(65) # A
type(chr(65)) # str
chr(240) # ð

# str functions
"python".upper() # PYTHON
"python".capitalize() # Python
" python ".strip() # python
("hello" "world") # helloworld
"hello"[0] # h
"hello world".split() # list [hello, world]
# python string are immutable!

# string/array slicing
"hello"[0] # h
"hello"[-1] # o
"hello"[:] # hello
"hello"[1:] # ello
"hello"[:-1] # hell 

# str formatting
"{lang} is awesome".format(lang="Python") # Python is awesome
lang = "Python"
f"{lang} is awesome" # Python is awesome

# strings are enumerable
upper = ""
for c in "hello":
    upper += c.upper()

# list comprehension on string
"".join([c.upper() for c in "hello"])

# byte conversion
"hello".encode() # b"hello"
"ð".encode() # conver into bytes b'\xc3\xb0'
len("ð") # 1
len("ð".encode()) # 2

Collection - list, tuple, set, dict

# list aka dynamic array
evens = [2, 4, 6]
evens.append(8) # [2, 4, 6, 8]
evens.remove(2) # [4, 6, 8]
evens.index(6) # 1
evens.extend([8, 10]) # [4, 6, 8, 8, 10]
evens.sort() # sort asc order
sorted(evens, reverse=True) # sort desc order

# looping a list
for val in range(5):
    print(val)

# looping list with index
for i, val in enumerate(range(5)):
    print(i, val)

# 2d array, 2x3 dimension
grid = [[1,2,3], [4,5,6]]
[0] * 3 # [0, 0, 0]
[[0, 1]] * 3 # [[0, 1], [0, 1], [0, 1]]

# jagged array
jagged = [[1], [1,2], [1,2,3]]

# tuple - immutable
immutable_list = tuple(2, 4, 6)

# set
distinct = set("AABBCCDD") # {'A', 'B', 'C', 'D'}

# conversion
a_list = list(range(5))
a_tup = tuple(a_list)
a_set = set(a_list)

# dict - map
hash_map = {"state": "GA", "post_code": 30342}
hash_map.keys() # dict_keys(['state', 'post_code'])
hash_map.values() # dict_values(['GA', 30342])
hash_map.get('state') # GA
hash_map.get('unknown_key', 'default_val') # default_val
hash_map.setdefault('foo', 'bar')
hash_map['foo'] = 'baz'

# looping a dict
for k, v in hash_map.items():
    print(k, v)

List comprehension

[x**2 for x in range(5)] # [0, 1, 4, 9, 16]
# set
{x**2 for x in range(5)} #  {0, 1, 4, 9, 16}
[key for key in hash_map.keys()] # ['state', 'post_code']
[val for val in hash_map.values()] # ['GA', 30342]
[item for item in hash_map.items()] # [('state', 'GA'), ('post_code', 30342)]
# dict
{key: key ** 2 for key in range(5)} # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Dates - datetime, time, timedelta

from datetime import date, datetime, timedelta
import time

# just the day
today = date.today()
# specific day 
dob = date(2000,1,1)
dob.isoformat() # '2000-01-01'
delta_days = today - dob # days
dt = datetime.fromisoformat('2011-11-04') # only python >= 3.7

# date diff using timedelta
yesterday = today - timedelta(days=1)
tomorrow = today + timedelta(days=1)

# datetime, it uses timezon
today = datetime.today()

# timestamp
timestamp = time.time()
today = date.fromtimestamp(timestamp)

# date formatting
today.strftime("%Y-%m-%d %H:%M:%S") # 2021-01-30 16:01:20

Functional programming - itertools, functools

# generator expressions - surrounded by parenthesis ()
iter = (line.strip() for line in ["hello world", "foo bar"])

# map, filter
list(map(lambda x: x.upper(), ["hello", "world"])) # ['HELLO', 'WORLD']
list(filter(lambda x: x % 2 == 0, range(10))) # [0, 2, 4, 6, 8]
# any, all
any([0,0,1]) # True
all([1,1,1]) # True

# itertools module
from itertools import count, cycle, repeat, takewhile, accumulate, groupby
import operator
# infinite iterator
# start counting from 100, 200, 300 ...
for i in count(100, 100):
    if i >= 1000:
        break
    print(i)

# takewhile 
list(takewhile(lambda x: x < 1000, count(100, 100)))

# cycle
for val in cycle("ABC"):
    print(val)

# repeat some numbers
repeat("ABC") # repeat infinitely
repeat("ABC", 3) # repeat 3 times

# running sum
list(accumulate(range(10)))

# group by - iterms need to be sorted!
{k: len(list(g)) for k, g in groupby('AABBBCCDEEE')} # {'A': 2, 'B': 3, 'C': 2, 'D': 1, 'E': 3}

# functools module - reduce
from functools import reduce
import operator
# sum up, init 0
reduce(operator.add, range(10), 0)

Testing in Go

Shaikhul Islam — Thu, 12 Nov 2020 04:57:01 +0000

Go testing support is awesome. It has a subcommand go test that looks for *_test.go files in package folder and run all test functions that starts with Test* prefix and report success or error.

All test functions should follow this naming convention

func TestXXXX(t *testing.T) {

}

t param is used to report success/failure messages. Some useful methods are

t.Errorf    // to report an error
t.Logf      // to log something
t.Skip      // skip a test

Pointers in Go

Shaikhul Islam — Sat, 07 Nov 2020 17:34:01 +0000

Go is a call-by-value language, meaning it will use a copy of each function arguments in function calls, if we want to update a variable we need to pass its address using a pointer.

Go pointer syntax are similar as C.

Pointer basics
Get address of a variable using & operator
Derefence using pointer * operator

// int variable
year := 2020
// we can get address using & op
year_add := &year
fmt.Println("Val of year: ", year)
fmt.Println("Address of year: ", &year)
fmt.Println("Val of year using pointer: ", *(&year))

Function call by value

func call_by_val(year int) {
  year = year + 1
}
call_by_val(year)
// year won't be updated as it uses a copy in call_by_val function
fmt.Println("Call by val res: ", year)

Function call by reference

func call_by_ref(year *int) {
  *year = *year + 1
}
call_by_ref(&year)
// here it will update the variable as we pass the address of the variable
fmt.Println("Call by ref res: ", year)

Method with Pointer Receiver
When using pointer receiver in methods, Go compiler does implicit conversion to match the type of receiver argument with receiver parameter.
To explain different use cases I will use a Point type and some methods as follows

type Point struct {
  X, Y int
}
func (p *Point) Scale(factor int) {
  p.X = p.X * factor
  p.Y = p.Y * factor
}
// pointer receiver param
func (p *Point) Print() {
  fmt.Println("X: ", p.X, ", Y: ", p.Y)
}

// receiver param of type Point, any change won't be reflected on receiver argument
func (p Point) ScaleVal(factor int) {
  p.X = p.X * factor
  p.Y = p.Y * factor
}

// value receiver param
func (p Point) PrintP() {
  fmt.Println("X: ", p.X, ", Y: ", p.Y)
}

Use case 1: both uses same pointer type No implicit conversion

p := Point{X:2, Y:3}
p_ptr := &p
Scale(p_ptr, 2)
// this will print the scaled point
Print(p_ptr)

Use case 2: receiver param is of type pointer

// receiver arg is of type Point, compiler implicitly get &p to match type with receiver param
p.Scale(2)
p.Print()

User case 3: receiver argument is of type pointer

// receiver param is of type Point but receiver arg is a pointer, compiler dereference the receiver but any mutation won't be reflected on receiver arg
p_ptr.ScaleVal(2)
p_ptr.PrintP()

Here is the full example in repl.it

Function Pointer in C

Shaikhul Islam — Thu, 05 Nov 2020 22:11:45 +0000

Taken from my personal blog post

Pointers in C are powerful and confusing at the same time. As we can point a pointer to structs, strings or arrays, we can point to function.

Signature: void (*POINTER_NAME) (args)

Say for example we have a function add that adds two number int add(int a, int b)

int add(int a, int b) {
    return a + b;
}

int mul(int a, int b) {
    return a * b;
}

The function pointer for this would be

int (*fun_ptr) (int, int)

We can assign this pointer to any function that accepts two int params and return an int.

To point to add function we can just assign it to the fun_ptr pointer as follows.

Now we can use the function pointer as follows

// assign it to add function
fun_ptr = add;
printf("2 + 3 = %d\n", fun_ptr(2, 3));

// assign it to mul function
fun_ptr = mul;
printf("2 * 3 = %d\n", fun_ptr(2, 3));

Now if we want to use function pointer as a function param we can use it directly or use a typedef as follows

// it takes a function pointer as param
int do_op(int a, int b, int (*ptr) (int, int)) {
  return ptr(a, b);
}
printf("2 + 3 = %d \n", do_op(2, 3, add));
printf("2 + 3 = %d \n", do_op(2, 3, mul));

As writing the function signature as a function param is hard, we can use the typedef to give it a new name

// using typedef
typedef int (*op_ptr_type) (int, int);
int do_op2(int a, int b, op_ptr_type ptr) {
  return ptr(a, b);
}
printf("2 + 3 = %d\n", do_op2(2, 3, add));
  printf("2 * 3 = %d\n", do_op2(2, 3, mul));

Here is the complete example in repl.it

Now, lets see how qsort takes advantage of function pointer.

The method signature of qsort is

void qsort(void *ptr, size_t count, size_t size, int (*comp)(const void *, const void *));

Here we can see the fourth param is a callback which is used to do custom comparison. If we want to sort some numbers in ascending order we need to define the comp function first using the same signature

int comp_int_asc(const void* a, const void* b) {
    // dereference to int
    int val1 = *(const int*)a;
    int val2 = *(const int*)b;

    if (val1 < val2) {
        return -1;
    } else if (val1 > val2) {
        return 1;
    }
    return 0;
}

we can use it in the qsort function as follows

int nums[] = {5, 1, 3, 9, 10, 8, -5, 0};
int int_size = sizeof(int);
int arr_len = sizeof(nums) / int_size;
// qsort is defined in stdlib
qsort(nums, arr_len, int_size, comp_int_asc);

Here is the full qsort example

How to start contributing in open source project

Shaikhul Islam — Thu, 28 Feb 2019 02:19:11 +0000

Here are my stories of getting involved in open source projects.

January 2017

I was frustrated with the almost no doc of hamcrest-php matchers, each time I wanted to use a matcher I have to look through the code, some of my office colleagues were also frustrated, I initially wrote down some matchers example and later thought this would be a great fit to add it in the doc, I updated the README file, open a PR and it eventually got merged!

March 2017

We use Phake mocking library heavily in our tests. In one case I found a method isn't working as expected in this issue, I opened a PR and added a comment in the issue. Even though I opened the PR back in March 2016, it actually got merged in March 2017!

June 2017

This time add test coverage on PHP source! I attended a php conference in Atlanta, GA on May, 2017. Inspired by this talk, I opened a PR that added tiny test coverage, it got a vetting before getting merged!

My recommendations for future open source contributors

Start with documentation, even if fixing a typo! Best way to start is look into your favorite framework/library/packages/whatever and see how you can contribute.
Check existing issues to see if there is anything that you can work on.
You don't need to be ninja/expert to contribute! There are places for everyone to contribute something/somehow that would help others.

Believe yourself and start contributing!

Code Review - let's do more!

Shaikhul Islam — Sun, 24 Feb 2019 07:17:25 +0000

Today I am going to discuss all the What/Why/Who/How questions about code review from my own experiences.

What?

You wrote some code and want someone from your team to take a look before pushing it into production.
In git workflow, developer write some code, push his/her changes and then open a pull request (PR) to let others review his PR.
It's becoming industry standards in CI/CD pipeline, even some companies wont let you merge code without getting your code reviewed. Here you can find dev.to all open PRs

Why?

To get feedback from peers
- Design principles, code structures etc.
Find potential bugs early
- Does it do what it's supposed to do?
Sanity check about your changes
- Any potential typo (also covered by static analysis tools if supported)
This is the right way to add any changes into production codebase!

Who?

Anybody can review anybody's code in your team unless otherwise defined by the team.
Active reviewer - someone who was assigned for review.
Passive reviewer - anybody else who is interested with the underlying codebase, can ask questions/clarifications about the changes.

Where?

Online - Github, Atlassian Crucible, Azure Devops and many others support code review functionality.
IM - yes! for sanity check you can provide the code snippet on Slack/Hipchat or whatever IM you are using.
Offline - Pair reviewing, ask your peer to review your code!

How?

Look for working code over perfect code.
New feature/Bug fix:
- Check for functionalities,
- Check for code structures does it follow the coding standards/guidelines?
- Is there any side effects for the given change?
New/Dark code:
- Check for functionalities
Test coverage?
- Easier for reviewer to understand how underlying code works.
- Boost confident to the code.

Code review best practices

Focus on the code, not the developer!
- This will remove any sort of bias that you may have while reviewing other's code.
Do not assume anything!
- Ask for clarity if you have any confusion.
If you have opinions/better alternatives, provide it as a feedback and ask what the code owner think about your thoughts.
Discuss explicitly the good/bad/side effects about the changes.
Resolve conflicts with positive attitude.
Appreciate the developer for what he/she is doing.

PR Best practices

Keep your PR small
- Small line of code means less bugs.
- Easier to review.
Use explicit/descriptive commit message
Provide explicit description in the PR, any todo or out of scope functionalities.
Provide test coverage, at-least areas that are important.
Be prepared to take negative feedback
- Address it as complement. Don't take anything too personal.
- Try to resolve the conflict as possible.
- Add the discussion/resolution in the commit message for posterity.
and last but not least, use emojis!

Here is an example PR of mine in PHP-src codebase that has almost everything I've said above!

Code review is a great way to learn from others, be part of it and make it a habit.

Cover image credit: https://c1.staticflickr.com/3/2203/2245445147_ff54c5997d.jpg

Hamcrest - make your test assertion more declarative

Shaikhul Islam — Fri, 22 Feb 2019 05:15:45 +0000

Hamcrest framework was first introduced in Java world, later implemented in other languages (ex. PHP, Python, Ruby) with some exceptions depending on language scopes. It provides a wide range of matcher objects which make test assertion more meaningful and easier to understand.

I am going to use Hamcrest's PHP implementation.

Here is a starter example.

Installation

Hamcrest-php can be installed using composer

// first thing first, install hamcrest-php and include it, your vendor location may vary.

require __DIR__ . '/vendor/autoload.php';

Quick use case - `assertThat`

Here is a simple example of assertThat matcher.

$result = true;
$expected = true;

assertThat($result, equalTo($expected));

Give a nice optional description

You can give it a nice description about the assertion.

assertThat("Result should be true", $result, equalTo($expected));

Add a decorator/sugar - `is`

Hamcrest also provides is decorator/sugar to make it super declarative!

assertThat("Result should be true", $result, is(equalTo($expected)));

Asserting negation - `not`

Sometimes we need to test negativity, hamcrest cover this with not matcher.

$expected = true;
assertThat("Result should not be false", $result, is(not(equalTo($expected))));

When it fails the assertion it shows error message like this

Hamcrest\AssertionError with message 'Result should be true
Expected: <false>
     but: was <true>'

Feel free to checkout other matchers or hamcrest in other supported languages!

Python data model revisited

Shaikhul Islam — Sun, 17 Feb 2019 06:17:05 +0000

Python's double underscore (aka dunder) methods (ex. __len__) aren't magic method but more than that. Each of python's built-in functions has a corresponding dunder method. Officially they are documented under python data model. Try google python data model, first result would point to python docs

So whats the big deal? Ever wonder why python's len method accept almost anything?

>>> len(range(5))
5

Now imagine a simple class like this

>>> class Foo:
...     pass
...

What would it return if I provide a Foo instance to len method?

>>> f = Foo()
>>> len(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'Foo' has no len()

It throws an exception TypeError complaining Foo has no len. Right, if we implement a __len__ on Foo object and return anything, python's len will look for a corresponding __len__ method for that object.

>>> class Foo:
...     def __len__(self):
...             return 5
...
>>> f = Foo()
>>> len(f)
5

boom! now it works and return what we want it to return by implementing the __len__ method.

Other most common dunder methods are

String representation

In [11]: class Point:
    ...:     def __init__(self, x, y):
    ...:         self.x = x
    ...:         self.y = y
    ...:
    ...:     def __str__(self):
    ...:         return 'Stringified Point: ({x}, {y})'.format(x=self.x, y=self.y)
    ...:
    ...:     def __repr__(self):
    ...:         return 'Representing Point: ({x}, {y})'.format(x=self.x, y=self.y)
    ...:

In [12]: p = Point(2,3)

In [13]: p
Out[13]: Representing Point: (2, 3)

In [14]: str(p)
Out[14]: 'Stringified Point: (2, 3)'

Operator overloading

We can implement binary arithmetic operations (ex +, - etc) implementing specific dunder methods.

In [21]: class Point:
    ...:     # continuing from previous example   
    ...:     def __add__(self, other):
    ...:         return Point(self.x + other.x, self.y + other.y)
    ...:

In [22]: p1 = Point(1, 2)

In [23]: p2 = Point(3,5)

In [26]: p3
Out[26]: Representing Point: (4, 7)

To learn more about emulating numeric types checkout the doc

Object comparison

We can implement __eq__ to compare between objects.

In [36]: class Point:
    ...:     # continuing from previous example
    ...:     def __eq__(self, other):
    ...:         # custom comparison
    ...:         return self.x == other.x and self.y == other.y
    ...:
In [37]: p1 = Point(2, 3)

In [38]: p2 = Point(2, 3)

In [39]: p1 == p2
Out[39]: True

In [40]: p3 = Point(2, 4)

In [41]: p1 == p3
Out[41]: False

To check other object customization please check the doc

Object attribute access

In [44]: class Point:
    ...:     # continuing from previous example
    ...:     def __getattr__(self, attr):
    ...:         if (attr == 'X'):
    ...:             return self.x
    ...:         elif (attr == 'Y'):
    ...:             return self.y
    ...:         else:
    ...:             raise AttributeError('Point object has no attribute {attr}'.format(attr=attr))
    ...:
In [45]: p = Point(2, 3)

In [46]: p.x
Out[46]: 2

In [47]: p.X
Out[47]: 2

In [48]: p.Xx
--------------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-48-9a49f22e192d> in <module>
---------> 1 p.Xx

<ipython-input-44-f5f17ce1e8d6> in __getattr__(self, attr)
     23             return self.y
     24         else:
--------> 25             raise AttributeError('Point object has no attribute {attr}'.format(attr=attr))
     26

AttributeError: Point object has no attribute Xx

More attribute access can be found in the doc

Emulating container/collection

We can also treat objects as collection and do all sort of function call that accept a collection.

In [49]: class Point:
    ...:     # continuing from previous example
    ...:     def __getitem__(self, index):
    ...:         if index == 0:
    ...:             return self.x
    ...:         elif index == 1:
    ...:             return self.y
    ...:         else:
    ...:             raise IndexError('No item found with index {index}'.format(index=index))
    ...:
In [50]: p = Point(2, 3)

In [51]: p[0]
Out[51]: 2

In [53]: p[1]
Out[53]: 3

In [54]: p[2]
--------------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-54-21c545b39f61> in <module>
---------> 1 p[2]

<ipython-input-49-6dd2485f38f5> in __getitem__(self, index)
     31             return self.y
     32         else:
--------> 33             raise IndexError('No item found with index {index}'.format(index=index))
     34

IndexError: No item found with index 2

Checkout the doc for whole list of dunder methods to emulate collection.

The best thing of python data model is that programmers hardly need to call the dunder method (except __init__ and few other customization methods), python interpreter will call these methods as needed.

I have put the complete Point class in this gist, feel free to play with all other dunder methods and share what you have learned with us.

TIL:When n + 1 == n in python?

Shaikhul Islam — Fri, 15 Feb 2019 03:48:15 +0000

Today I was looking to python docs for doctest and found following

if n+1 == n:  # catch a value like 1e300
    raise OverflowError("n too large")

Immediately I opened the shell and try it:

>>> from math import exp
>>> n = exp(300)
>>> n
1.9424263952412558e+130
>>> n + 1 == n
True

TIL n+1 == n! and how to guard against a ridiculously large number

Spy PHP process with gdb + php5-dbg

Shaikhul Islam — Tue, 29 Jan 2019 04:15:54 +0000

In my previous post I briefly mentioned debugging a php process with gdb. In this post I am going to describe the steps and required tools to successfully get insights/stacktrace from a running PHP process.

First thing first, make sure you have install php5-dbg package. In this example I am using an Ubuntu box.

$ dpkg-query -l | grep php5-dbg

If not found install it first

$ sudo apt-get install php5-dbg

We also need a .gdbinit file. Get it from here according to php version.

Now it's time to start gdb with the php process.

$ sudo gdb -p PID

At the point gdb will pause the program, you need to type continue inside the gdb shell.

> cont

Wait for a while and then press Ctrl+C to get back to gdb shell.

Now source the .gdbinit, I put it into my home directory.

> source ~/.gdbinit

Type zbacktrace to see actual backtrace from the underlying php program.

> zbacktrace

After getting the stacktrace detach the program from gdb and then quit, if you dont detach the program, gdb will kill it when you quit gdb.

> detach
> q

You can check other commands available in the .gdbinit file.

This helped me today to get actual stacktrace of the php program instead of php interpreter calls that I got with bt command.

gdb is really an awesome sauce!

How to spy a process

Shaikhul Islam — Sat, 26 Jan 2019 01:44:33 +0000

Have you ever stuck with a program running for a long time or wondering what the process is doing? No worries! You are not alone, same thing happen me a lot.

Most recently I have to spy on a long running php process to figure out why/where its taking so much time.

Here is my approach to spy any process in linux environment.

Find out the process ID by running a ps command and grep for specific program. In my case it was a php program so I type following:

$ ps aux | grep "some_program.php"

Then strace the process. You need root access.

$ strace -p PID

Don't worry if it dumps a lot in the standard output. It trace all the system calls (read,write,open,close,poll,recvfrom etc) that the process is doing. Look for any interesting pattern.

To see which files are opened by the process you can try lsof command. This helps me to figure out which file paths are used by the program.

$ lsof -p PID

To further debug the process try gdb.
First, attach the program with gdb.

$ gdb -p PID

Now you are in gdb shell. type cont and wait for a while. and then hit ctrl + c to stop it.

> cont

It will stop and return back to gdb shell. Now type bt it will dump the backtrace. I can see some php library function calls.

> bt

These few commands helps me a lot to deep dive into a process and figure out whats going on. This is specially helpful when you don't know much about the underlying codebase.

To learn more about strace check out following two articles.

strace zine by Julia Evans
What is php doing by Derick Rethans