DEV Community: Triet Nguyen

10 Simple Ways To Speed Up Your Python Code

Triet Nguyen — Fri, 10 Nov 2023 07:26:10 +0000

Why should you care about your Python code performance?

Let's be honest first, you're not using Python in your project because it's a fast programming language, but because it's easy for you to translate your idea into real life working product. So from the beginning, speed is not a problem for you, so why should you even bother to spend your precious time upgrading it? For me, writting code is not just putting sloppy working pieces together and then call it a day. I want to learn more about how to write faster, cleaner and simpler code. Because of this, I've been looking around for ways to increase my Python code's performanc ewithout sacrificing its readability. Let me show you the way of the force!

1. Choose wisely between "Ask for Forgiveness" and "Look before you leap"

Ask for forgiveness: You run your code like normal, then wrap them in the try/catch block.

try:
    with open("path/to/file.txt", "r") as fp:
        return fp.read()
except IOError:
    # Handle the error or just ignore it

Look before you leap: Try to check everything possible that can result in a bug or crash in you code before you run it.

# Check if file exists and if we have the permission to read it
if Path("/path/to/file").exists() and os.access("path/to/file.txt", os.R_OK):
    with open("path/to/file.txt") as input_file:
        return input_file.read()

When to use which: In most cases, "Look before you leap" is slower than "Ask for forgiveness" (~30-80% slower). But only in cases where you expect your code to fail alot, then "Look before you leap" should be faster (4x faster). Because handling exceptions is very expensive

2. How to properly filter a list:

There are 3 popular ways to filter a list in Python:

For Loop: This is the basic way to get your job done, it's easy to read and understand.

# Get all even numbers in the list contains 1 million number
def filter_odd_number():
    lst = list(range(1000000))
    res = []
    for item in lst:
        if item % 2 == 0:
            res.append(item)
    return res

List Comprehension: Your code will be drastically faster (~50% faster) than the normal for loop, and shorter as well. It's faster than the for loop because when you use a for loop, on every iteration, you have to look up the variable holding the list and then call its append() function. This doesn't happen in a list comprehension. Instead, there is a special bytecode instruction LIST_APPEND that will append the current value to the list you're constructing.

def filter_odd_number():
    lst = list(range(1000000))
    return [item for item in lst if item % 2 == 0]

Python built-in filter() function: This function will return a generator rather than the whole list at once. If you want to get the whole list at once, this should be the worst performance-wise option.

def filter_odd_number():
    lst = list(range(1000000))
    return filter(lambda x: x % 2 == 0, lst)

When to use which: Most of the time, list comprehension is the winner, it's faster, shorter and some may say it looks cooler as well. But it has 1 limitation is that you cannot pact multiple statements inside of it, but don't worry, you can always wrap your logic inside a function and use it inside a list comprehension. But if you have a realy big list (I'm talking billions of item) then consider using filter(), you definitely don't want your big list to be duplicated.
Note: Dictionary comprehension is also faster than normal for loop for many tasks

3. Checking for True or False

We have 3 ways to achieve this:

if var == True: This is bad, ~120% slower than the winner
if var is True: This is also bad, ~60% slower than the winner
if var: Good, recommended by PEP8
The winner is clearly if var

Wait, it's not going to be that easy. In Python, there's a concept called "truthy & falsy", it means that there are variables interpreted as True or False if you run bool(variable) (explicitly by you or by the interpreter when you put them behind the if clause). Now, you may have different logic to deal with a certain type of the data.

var = [] # or None or True
if len(var) == 0:
    print("empty")
elif var is None:
    print("null")
elif var is True:
    print(true)

To check if a variable is equal to True/False (and you don't have to distinguish between True/False and truthy / falsy values), use if variable or if not variable. It's the simplest and fastest way to do this.
To check that a variable is explicitly True or False (and is not truthy/falsy), use is (if variable is True).
To check if a variable is equal to 0 or if a list is empty, use if variable == 0 or if variable == [].

4. Use `in` to check if item exists in a collections of item

in in a powerful tool to check if an item is in a list/set/tuple or a dictionary (use in would only check if a Key exists in a dictionary). Do not use for to do this kind of task (~50% slower)
Note: If you need to "check for membership" a lot, then you should consider using dict or set to store your items, because they have constant average lookup time.

5. Remove duplicate value inside a `list`

There should be many ways to implement this task, but 3 stand out the most:

Use for loop or list comprehension to manually remove duplicate value: Slow and takes longer to implement
Use list(set(arr)) to do the trick: This is ths fastest and shortest way as well, too bad they're going to shuffle the order of your original list (if you don't care about it then good!)
Use list(dict.fromkeys(arr)) for Python 3.6 and above or list(OrderedDict.fromkeys(arr)): This code is a little slower than the set way but it will keep your original list order

6. Declare a data structure in style

When you need to declare an empty list, dict, tuple or set, you should use the Pythonista syntax rather than explicitly call the function because it will be faster and shorter to (~100% faster). This is because when you call the function, Python would then try to check if there's any other function with that name in your script before thinking about using its own

List: Use [] instead of list()
Dict: Use {} instead of dict()
Tuple: Use () instead of tuple()
Set: Use {x,} instead of set([x])

7. String format

Python 3.6 introduce the famous f-string for us to build formated string (before that we have to use the .format() or Template string) and since then it's hard to find someone who doesn't love them! But is it the optimal way to create a format string?

Yes, f-string perform consistently better than its competitor, here's an example

name, age = "Triet", 24
s = f"My name is {name}, I'm {age} years old"

The "concat string" is also an honorable mention, but its performance is as consistent as the f-string (sometimes it's even faster than the f-string)

8. Map vs List Comprehension

In the previous section, I've said that you should always use "List comprehension" instead of for loop for tasks that require you to loop over a list. One big problem with that is "List comprehension" cannot take too many logic inside (without ruin your beautiful code), so my solution back there is to wrap that logic into a function, then call it inside of the "List comprehension". It's a good work around, But should you do it?

No you should not! Use map() instead. map() is faster (~50%) and easier to write as well. Here's an example:

def pow2(x):
    return x**2

arr = list(range(1000000))

res = list(map(pow2, arr))

Why we need the list() outside of map() you asked? Well map() return a generator which would return item one by one when you call it, so list should get the whole result and put them back inside a list

9. Inlining Functions

This is one way to increase your code performance that go against the concept of beautiful code - Try to wrap the whole function's logic in just one line. And yes, it may sound silly at first but it's actually work (~100% faster)! Here's an example

def inline_functions():
    return sum([sum([sum([1 for _ in range(100)]) for _ in range(100)]) for _ in range(100)])

I've won... But at what cost?

This should only be done with function that has so little logic inside! please don't hurt your colleague.

10. Upgrade your Python version

Yes, you heard me, go upgrade your Python version, it's free and can affect instantly! Python 3.11 is 10-60% faster than Python 3.10

Reference

Sebastion Witowski's blog: https://switowski.com/blog

Thank you for reading the article!

Golang Wikipedia module

Triet Nguyen — Wed, 02 Nov 2022 14:00:00 +0000

Wikipedia is the largest and most-read reference work in history. It was ranked the 7th most popular site in 2022 and hosted by the Wikimedia Foundation, an American non-profit organization funded mainly through donations.

Being one of the best free online sources for information, Wikipedia gave you APIs to fetch the data yourselves. But learning how to use those APIs may take weeks or months to master. In this article, we'll see how to use Golang API wrapper Go-wiki to access and fetch a variety of information from the Wikipedia website.

github.com/trietmn/go-wiki Go-wiki is a Golang Wikipedia API wrapper - The Golang module that makes it easy to access and parse data from Wikipedia.

Instalation

To install Go-Wiki package, you need to install Go and set your Go workspace first.

You first need Go installed, then you can use the below Go command to install Go-wiki.

go get -u github.com/trietmn/go-wiki

Import it in your code.

import "github.com/trietmn/go-wiki"

Examples

1. Getting the summary of any title

Code:

package main

import (
    "fmt"
    "github.com/trietmn/go-wiki"
)

// Getting summary of Wikipedia page "Rafael Nadal"
func main() {
    res, err := gowiki.Summary("Rafael Nadal", 5, -1, false, true)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("Summary: %v\n", res)
}

Output

Summary: Rafael Nadal Parera (Catalan: [rəf(ə)ˈɛl nəˈðal pəˈɾeɾə], Spanish: [rafaˈel naˈðal paˈɾeɾa]; 
born 3 June 1986) is a Spanish professional tennis player. He is currently ranked world No. 2 in singles by the
Association of Tennis Professionals (ATP). He has been ranked world No. 1 for 209 weeks, and has finished as the year-end No.

2. Searching title and suggestions

Code:

package main

import (
    "fmt"
    "github.com/trietmn/go-wiki"
)

// Search and get suggestion
func main() {
    // If you don't need the suggestion, set the 3rd parameter to false
    search_result, suggestion, err := gowiki.Search("nadal", 3, true)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("Search result: %v\n", search_result)
    fmt.Printf("Suggestion: %v\n", suggestion)
}

Output:

Search result: [Rafael Nadal Federer–Nadal rivalry Nadal (surname)]
Suggestion: nasal

3. Getting the Wikipedia page information

Code:

package main

import (
    "fmt"
    "github.com/trietmn/go-wiki"
)

// Getting the Wikipedia page "Rafael Nadal"
func main() {
    // Get the page
    page, err := gowiki.GetPage("Rafael Nadal", -1, false, true)
    if err != nil {
        fmt.Println(err)
    }

    // Get the content of the page
    content, err := page.GetContent()
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("This is the page content: %v\n", content)
}

Output:

This is the page content: Rafael Nadal Parera (Catalan: [rəf(ə)ˈɛl nəˈðal pəˈɾeɾə], Spanish: [rafaˈel naˈðal paˈɾeɾa]; born 3 June 1986) 
is a Spanish professional tennis player. He is currently ranked world No. 2 in singles by the Association of Tennis Professionals (ATP). 
He has been ranked world No. 1 for 209 weeks, and has finished as the year-end No. 1 five times. Nadal has won an all-time record 22 Grand 
Slam men's singles titles, including a record 14 French Open titles. He has won 92 ATP singles titles, including 36 Masters titles, with 63 
of these on clay courts. Nadal is one of only two men to complete the Career Golden Slam in singles. His 81 consecutive wins on clay is the 
longest single-surface win streak in the Open Era...

Note: The above code will get you the content of the page, if you want to get other information, use other method like:

Methods	Description	Example
Equal	Check if 2 pages are equal to each other	page1.Equal(page2)
GetContent	Get the page content	page.GetContent()
GetHTML	Get the page HTML	page.GetHTML()
GetRevisionID	Get revid field of a page	page.GetRevisionID()
GetParentID	Get parentid field of a page	page.GetParentID()
GetSummary	Get the summary of the page	page.GetSummary()
GetImagesURL	Get all of the image URL appear in the page	page.GetImageURL()
GetCoordinate	Get the page coordinate if exist	page.GetCoordinate()
GetReference	Get all of the extenal links in the page	page.GetReference()
GetLink	Get all the titles of Wikipedia page links on a page	page.GetLink()
GetCategory	Get all of the categories of a page	page.GetCategory()
GetSectionList	Get all of the sections of the page	page.GetSectionList()
GetSection	Get the content of a specific section in the page	page.GetSection("History")

4. Changing the language

Code:

package main

import (
    "fmt"
    "github.com/trietmn/go-wiki"
)

// Getting summary of Wikipedia page "Rafael Nadal"
func main() {
    // Change the Wikipedia language to Vietnamese using the language prefix (ex: vi, fr, en, kr, cn)
    gowiki.SetLanguage("vi")

    // Get the summary in Vietnamese
    res, err := gowiki.Summary("Rafael Nadal", 3, -1, false, true)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("Summary: %v\n", res)
}

Output:

Summary: Rafael "Rafa" Nadal Parera (IPA: [rafa'el na'ðal], sinh ngày 3 tháng 6 năm 1986 tại Manacor, Mallorca) là một vận động viên quần vợt 
chuyên nghiệp người Tây Ban Nha hiện đang giữ vị trí số 3 thế giới. Nadal được đánh giá là một trong những tay vợt xuất sắc nhất mọi thời đại. 
Nadal đang nắm giữ kỷ lục 22 Grand Slam ở nội dung đánh đơn, cùng với đó là 2 huy chương vàng Olympic: đơn nam tại Olympic 2008 và đôi nam tại 
Olympic 2016, 36 chức vô địch ATP World Tour Masters 1000, 21 chức vô địch ATP Tour 500, 5 chức vô địch Davis Cup cùng đội tuyển Tây Ban Nha vào 
các năm 2004, 2008, 2009, 2011 và 2019 cùng nhiều danh hiệu khác.

5. Other functions that you may use

Functions	Description
Suggest	Get suggestion from your keyword
Geosearch	Use the Wikipedia Geosearch
GetRandom	Get some random Wikipedia page titles
GetAvailableLanguage	Get the list of Wikipedia supported language
SetMaxCacheMemory	Set the maximum amount of API response stored in the Cache
SetCacheDuration	Set the maximum "age" of API response stored in the Cache

Documentation

For further information, check out these sources:

Github: https://github.com/trietmn/go-wiki
GoDoc: https://pkg.go.dev/github.com/trietmn/go-wiki
Reddit Discussion: https://www.reddit.com/r/golang/comments/yi5nal/gowiki_the_golang_module_that_makes_it_easy_to/

DEV Community: Triet Nguyen

10 Simple Ways To Speed Up Your Python Code

Why should you care about your Python code performance?

1. Choose wisely between "Ask for Forgiveness" and "Look before you leap"

2. How to properly filter a list:

3. Checking for True or False

4. Use in to check if item exists in a collections of item

5. Remove duplicate value inside a list

6. Declare a data structure in style

7. String format

8. Map vs List Comprehension

9. Inlining Functions

10. Upgrade your Python version

Reference

Thank you for reading the article!

Golang Wikipedia module

Instalation

Examples

1. Getting the summary of any title

2. Searching title and suggestions

3. Getting the Wikipedia page information

4. Changing the language

5. Other functions that you may use

Documentation

4. Use `in` to check if item exists in a collections of item

5. Remove duplicate value inside a `list`