DEV Community: HackerCulture

The Ultimate Python Programmer Roadmap (2023)

HackerCulture — Tue, 10 Jan 2023 21:42:31 +0000

At HackerCulture , we're all about Python. Our mission is to get more people familiar with the Python programming language, and we teach you how to use python to write better programs.

Python is the most loved programming language by both beginners and experts, according to multiple surveys. And not only software developers but other people from different disciplines use Python for a variety of different tasks, such as data analysis and visualization, artificial intelligence and machine learning, automation, etc.

There has never been a better time to learn Python, and with our resources created by professional Python developers with decades of experience, you can learn the skills and techniques that really matter in the real world. You can use Python to build web apps, mobile apps, and desktop applications, as well as software testing and even some actual hacking.

Whether your goal is to learn python for web development, data science, or machine learning, we'll go over the most important libraries and resources to learn after getting the basics down.

Python Basics
Intermediate Python
Advanced Python Topics
Python for Web Development
Python for Automation
Python for DevOps
Python for Data Science
Python for Machine Learning
Tips for learning Python programming

Python Basics

First, we need to install python so we can play around with it:

Setting Up Python and Virtualenvs with Pyenv

Then, you can follow this roadmap to go from Python beginner to knowing "just enough to be dangerous" with Python. While this list doesn't cover every single feature, it does cover many of Python’s most noteworthy features and will enable you to read and write Python modules and programs.

Python Basics: Code Your First Python Program
Variables and operators
Numbers and math
Strings and string methods
Basic Data Types in Python
Loops in Python
Exception handling
Functions
Modules

Intermediate Python

This section will give you the tools to make complex applications and introduce you to concepts about Object Oriented Programming (OOP) in Python:

Advanced Python Topics

This section builds upon the previous ones and teaches you how to use frameworks & libraries available in the Python language:

At this point in time, once you have gone through all the previous steps, you should have everything you need to make use of Python in real-world applications.

Python is used in almost every industry, and you have a few career path options. Let's look at some of these.

Python for Web Development

The role of a web developer is to build websites and applications that are accessible using a web browser. Python is typically used on the backend side.

A popular web framework that many companies use to build web applications is Flask. This tutorial will help you become familiar with building apps with Flask:

Python REST APIs with Flask

You should also become familiar with databases. This tutorial covers how to integrate databases with your Flask server:

Connecting a Database to your Flask Project with SQLAlchemy

You can also learn how to deploy Flask apps so you can share them with the world in this article:

How to deploy your Flask apps

Finally, you may be interested in learning more about the web and how your users will interact with your apps on their browsers. This course will help you become more of a well-rounded full-stack web developer:

Web Design Course for Everybody

Python for Automation

Automation is the field of performing tasks that are repetitive or time-consuming, such as data entry, web scraping, and automation of various system processes, in a way that they can be reused.

A specific use case for automation is the continuous testing of software projects, in which we write scripts to interact with an existing program and report errors.

Getting started with testing in Python

Automation can also be used to control other software, such as a web browser, Excel, or even hardware, such as robotic systems. Some libraries worth checking out for this are:

Selenium
Beautiful Soup

Python for DevOps

DevOps engineers are responsible for implementing and maintaining infrastructure to support software systems, and they work to ensure that software systems are deployed quickly, efficiently, and with minimal downtime.

This book provides a comprehensive introduction to the field of DevOps:

Python for DevOps: Learn Ruthlessly Effective Automation

Some of the other technologies used by DevOps engineers to automate software deployments and maintain a smooth operation are:

Docker
Kubernetes

Python for Data Science

A data scientist's primary concern is to extract insights and knowledge from data. This typically involves a number of different tasks, including collecting and cleaning data, developing predictions and forecasts using statistics, and visualizing results.

A data scientist must be familiar with databases and able to query large amounts of data in order to identify patterns and trends. Usually, the main tools used by data scientists are:

If you want to get deeper into data science, consider the following books:

Data Science from Scratch

Python for Data Analysis: Data Wrangling with Pandas, NumPy, IPython

Python for Machine Learning

A machine learning engineer creates and develops computer programs that can "learn" from data and make predictions or decisions without being explicitly programmed to do so to solve real-world problems like predicting the weather or detecting facial expressions.

This typically involves working with large sets of data, which can be explored with the pandas library to prepare the data for use in models:

Getting started with Pandas in Python

A very popular library that helps with training machine learning models for different problems is Scikit-learn.

If you are interested in building other language applications like translating speech to text or summarizing a long article, these resources are for you:

Natural Language Processing (NLP) with Python - Tutorial

If you prefer to read a book instead, these books are an excellent way to get into machine learning with Python:

Introduction to Machine Learning with Python

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

Tips for learning Python programming

Learning how to write computer programs is hard, but learning Python, in particular, is significantly easier than many other programming languages. These tips will help you stay focused and motivated to achieve your goals:

Have patience. Learning a new language always requires more effort and time, so understand that it will take time to let everything sink in.
Don't feel intimidated by complex terms, errors, and issues. Don't give up, this happens to everyone in programming. Research the topic using other resources and come back to it with a clear mind.
Frustration and pain are a part of the process. Embrace them instead of avoiding them, and you will conquer them.
Be consistent. If you are not consistent in learning, it will take a lot more time and effort.
Don't try to learn everything at once. It's better to practice a specific topic every day before moving on to the next.
Build small projects to try out the things you learn and build up your confidence. You should add them to your github profile to help you land jobs.

Did you find this helpful?
I’d love to hear about it. Please let me know in the comments.

Do you have any questions?
Leave your question in a comment below, and we'll answer it with our best advice.

Python asyncio - A Guide to Asynchronous Programming

HackerCulture — Mon, 09 Jan 2023 22:00:13 +0000

In this guide, we'll introduce asynchronous programming in Python and review fundamental concepts like how to define, create and run asynchronous functions, coroutines, and some common use cases and best practices.

You may have seen asyncio in a new project you started working on. Or perhaps in a code example of how to perform a common task:

Perhaps you need to use an API and the code examples use asyncio.
Or you need to integrate a new library that uses asyncio.
Or you found some code snippets that use asyncio.

Once you understand the concepts in this guide, you will be able to develop programs that can leverage the asyncio library in Python to process many tasks concurrently and make better use of your machine resources, such as additional CPU cores.

Introduction to Asynchronous Programming
Getting Started with Asyncio in Python
Working with Asyncio Event Loops
What Are Coroutines
Using Asyncio for Network Programming
Advanced Asyncio Concepts
When to Use Asyncio
When Not to Use Asyncio
Best Practices for Asynchronous Programming in Python

Introduction to Asynchronous Programming

Asynchronous refers to the concept of not occurring at the same time as something else. In the context of programming, asynchronous refers to the execution of tasks that do not block the flow of execution of other tasks. This means that multiple tasks can be performed concurrently, rather than sequentially.

This is achieved by using asynchronous function calls, which return control to the calling function before the task is completed, allowing other tasks to be executed in the meantime.

For example, we can make an asynchronous function call. This will issue the request to make the function call and will not wait around for the call to complete. We can choose to check on the status or result of the function call later.

One of the main benefits of asynchronous programming is improved performance, as it allows for non-blocking IO operations and allows multiple tasks to be performed simultaneously.

Getting Started with Asyncio in Python

The asyncio module contains utilities to implement asynchronous programming in Python.

It allows you to write concurrent code using asynchronous programming techniques rather than traditional thread-based concurrent programming.

You may be wondering how asyncio differs from traditional thread-based concurrent programming in Python...

Asyncio is particularly useful in Python because it allows you to write concurrent code in a single-threaded environment, which can be more efficient and easier to work with than using multiple threads.

Understanding the async, await syntax

We use def async to define asynchronous functions in Python. These are special types of functions that don't execute right away when called. Instead, they are scheduled, and we must then use await to execute and wait for the function to return.

This last point is key and is the reason why async functions are so powerful.

We can also execute async functions in other parts of our program with asyncio.run()

async def an_async_function():
    await asyncio.sleep(1)
    print("Hello, World!")

asyncio.run(my_async_function())

Running async functions in Python with asyncio.run

What is the Event Loop

The event loop is the central mechanism in asyncio that schedules and runs asynchronous tasks. It works by continuously polling for events and running the appropriate tasks when they are ready.

You can think of the event loop as a kind of "coordinator" that manages the execution of asynchronous tasks.

Creating and customizing event loops

By default, asyncio will create an event loop for you when you use asyncio.run() or call asyncio.AbstractEventLoop.run_until_complete(). However, you can also create and customize your own event loop if needed.

loop = asyncio.get_event_loop()
loop.run_until_complete(an_async_function())
loop.close()

How to get the current event loop

The event loop starts with the loop.run_until_complete() method, which blocks until all tasks have completed. Finally, the event loop is closed with the loop.close() method.

Here are some other ways you can run an event loop in Python using the asyncio module:

loop.run_forever(): This method runs the event loop indefinitely. It blocks until the event loop is stopped with the loop.stop method or until an exception is raised.
loop.run_until_complete(future): This method runs the event loop until the given future is completed. It blocks until the future is done or until an exception is raised.
loop.run_in_executor(executor, func, *args): This method runs the given func in a thread or process executor and returns a Future object. The event loop will run until the Future is done or until an exception is raised.
loop.run_as_soon_as_possible(callback): This method schedules the given callback to be run as soon as possible. It does not block.
loop.call_later(delay, callback, *args): This method schedules the given callback to be run after the specified delay in seconds. It does not block.

What Are Coroutines

A coroutine is a task that can be suspended and resumed. In the context of asynchronous programming, when we talk about tasks that can be performed concurrently, we are referring to coroutines.

Therefore, coroutines are the unit of concurrency used in asyncio programs.

Many coroutines can be created and executed at the same time. They have control over when they will suspend and resume, allowing them to cooperate as to when concurrent tasks are executed.

This is called cooperative multitasking and is different from the multitasking typically used with threads called preemptive multitasking.

A coroutine can be defined using the async def expression. It can take arguments and return a value, just like a function. Calling a coroutine function will create a coroutine object, but it does not execute the coroutine function right away.

async def my_coroutine(delay):
    await asyncio.sleep(delay)
    print(f'Finished waiting for {delay} seconds')

# creates but does not execute the coroutine
coroutine_1 = my_coroutine(1)
# await starts the coroutine and blocks until finished
result = await my_coroutine(2)

Creating and awaiting coroutines

When a coroutine is created but never executed, we will see a warning like this when the Python program finishes:

sys:1: RuntimeWarning: coroutine 'my_coroutine' was never awaited

We can avoid this warning and support top-level await expressions by starting the Python interpreter with a default event loop. This is also the default in IPython.

python -m asyncio script.py

Having top level await is neat, but it can cause problems too. There can only be one event loop running in a thread. If any of your code tries to launch an event loop, perhaps by calling asyncio.run(), you'll get an error, so be careful.

What's so great about coroutines

Threads and processes achieve multitasking managed by the operating system (OS) that chooses which threads and processes should run, when, and for how long. The OS switches between threads and processes rapidly, suspending those that are not running and resuming those granted time to run. This is called preemptive multitasking.

Coroutines in Python provide an alternative type of multitasking called cooperating multitasking. This allows coroutines to cooperate by design, choosing how and when to suspend their execution.

Another key aspect of coroutines is that they are more lightweight than threads. This means they are faster to start and use less memory. They may use less than 1 KB of memory to execute. Essentially a coroutine is a special type of function, whereas a thread is represented by a Python object associated with an operating system thread with which the object must interact.

Coroutines are a new alternate, interesting, and powerful approach to concurrency, different from thread-based and process-based concurrency. As such, we may have thousands of threads in a Python program, but we could easily have tens or hundreds of thousands of coroutines just in one thread.

You can also schedule coroutines for execution in separate threads, with ThreadPoolExecutor, or separate processes using ProcessPoolExecutor.

How to Run Blocking Tasks with Asyncio

We often need to execute a blocking function call within an asyncio application because, in practice, most workloads include a mix of IO-bound operations and also CPU-bound operations.

This could be for many reasons, such as:

To execute a CPU-bound task like calculating something.
To execute a blocking IO-bound task like reading or writing from a file.
To call into a third-party library that does not support asyncio yet.

Making a blocking call directly in an asyncio program will cause the event loop to stop while the blocking call is executing. It will not allow other coroutines to run in the background.

This can be prevented by running the blocking call outside of the event loop, which we can do with asyncio.to_thread().

# execute a function in a separate thread
await asyncio.to_thread(my_blocking_task)

How to run blocking task with asyncio

The asyncio.to_thread() function takes a function to execute and any arguments. It returns a coroutine that can be awaited or scheduled as an independent task. The function is then executed in a separate thread.

The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls.

As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks, and we should not use this method of asyncio for CPU bound tasks.

Using Asyncio for Network Programming

Network calls are a good use case for asyncio since they're IO-bound, and we can just fire and forget the write or read_,_ and our program can go on to perform other tasks. The caller does not need to wait for the operation to complete before returning, so we should not waste time just checking on them.

The read and write operations are performed somehow (e.g., by the underlying operating system or perhaps a task queue like Kafka), and the status of the action and/or data is retrieved by the caller later, once available, or when the caller is ready.

Non-blocking IO is a way of performing IO where reads and writes are requested, although performed asynchronously. The caller does not need to wait for the operation to complete before returning. Non-blocking IO is implemented in practice with asynchronous programming.

Let's go through a few asyncio examples that implement non-blocking IO in client and server networking:

HTTP server

This is an example of an asyncio-based HTTP server that serves static files from a given directory. It uses the aiohttp library to handle HTTP requests and responses.

import asyncio
from aiohttp import web

async def handle(request):
    with open('index.html', 'rb') as f:
        return web.Response(body=f.read(), content_type='text/html')

async def main():
    app = web.Application()
    app.add_routes([web.get('/', handle)])
    return app

if __name__ == ' __main__':
    web.run_app(main())

Async web server in Python

WebSocket server

This is an example of an asyncio-based WebSocket server that broadcasts messages to all connected clients. It uses the websockets library to handle WebSocket connections.

import asyncio
import websockets

async def broadcast(websockets, message):
    for ws in websockets:
        await ws.send(message)

async def handler(websocket, path):
    async for message in websocket:
        await broadcast(websockets, message)

async def main(host, port):
    server = await websockets.serve(handler, host, port)
    await server.wait_closed()

if __name__ == ' __main__':
    asyncio.run(main('127.0.0.1', 8888))

Async websockets server and handlers

WebSocket client

This is an example of an asyncio-based WebSocket client that connects to a server, sends a message, and prints the response. It uses the websockets library to handle the WebSocket connection.

import asyncio
import websockets

async def main():
    async with websockets.connect('ws://localhost:8888') as websocket:
        await websocket.send("Hello, world!")
        response = await websocket.recv()
        print(f'Received: {response}')

asyncio.run(main())

Async websockets client connections

Advanced Asyncio Concepts

So far, we've only seen examples of how to run a single asynchronous function to perform non-blocking IO operations.

When we have many operations to run concurrently, we need to coordinate and manage those operations. Fortunately, the asyncio module has some functions that make it easier for us.

How to use asyncio.gather() to wait for multiple async tasks

import asyncio

async def task_one():
    print("Starting task one")
    await asyncio.sleep(1)
    print("Finishing task one")
    return 1

async def task_two():
    print("Starting task two")
    await asyncio.sleep(2)
    print("Finishing task two")
    return 2

async def main():
    # This will wait for all the coroutines
    results = await asyncio.gather(task_one(), task_two())
    print(results)

asyncio.run(main())

Using asyncio.gather to wait for coroutines

This code will produce the following output:

Starting task one
Starting task two
Finishing task one
Finishing task two
[1, 2]

As you can see, the tasks are run concurrently rather than sequentially. This can be useful for improving the performance of an application by allowing it to make better use of available CPU resources.

The asyncio.gather() method accepts many arguments and returns a list of the results returned by all the coroutines.

It is also common to create many coroutines beforehand and then gather them later. We can collect many coroutines together into a list either manually or using a list comprehension. We can then unpack this list as arguments to gather(). Note the use of the star operator (*) here:

# creates many coroutines
coros = [my_async_task(i) for i in range(10)]
...
# then, run the tasks
results = await asyncio.gather(*coros)

How to await a list of coroutines using asyncio.gather

How to use asyncio.wait() to manage multiple asynchronous tasks

The asyncio.wait() function blocks and returns a tuple containing two sets of tasks: those that have completed and those that are still pending. In this example, we iterate over the completed tasks and print their results.

Using the asyncio.wait() function, you can easily manage multiple asynchronous tasks and perform actions when they have completed. This can be particularly useful for situations where you need to wait for multiple tasks to complete before moving on to the next step of your program and for setting timeouts for some tasks.

# Run these tasks and wait 
tasks = [task_one(), task_two()]
    completed, pending = await asyncio.wait(tasks)

    results = [t.result() for t in completed]
    print(results)

Using asyncio.wait to manage async tasks

Alternatively, we could use the asyncio.as_completed() function to run concurrent tasks and iterate their results as they become available. This is possible because asyncio.as_completed() returns an iterable of the coroutines that can be awaited.

Chaining multiple asynchronous tasks

There are situations where you may need to perform a series of asynchronous tasks in a specific order. Since coroutines are awaitable, another coroutine can await it. You may use this class to chain async tasks since the asyncio library does not (yet) provide a similar function.

class Chain:
    "A class to execute async tasks sequentially"
    def __init__ (self, tasks):
        self.tasks = tasks

    async def __call__ (self, *args, **kwargs):
        for task in self.tasks:
            args = await task(*args, **kwargs)
            args = (result,)
        return result

The Chain class executes async tasks sequentially

To use this class, you can pass a list of asynchronous tasks to the constructor and then call the Chain instance to execute the tasks in the specified order. The result of each task is passed as an argument to the next task in the chain.

chain = Chain([task_one, task_two])
# block until tasks are performed synchronously
result = await chain() # returns the last task result

Running tasks in sequence with Chain

When to Use Asyncio

Coroutines are an alternative to threading-based concurrency that is provided by the Python language and further supported by the asyncio module. They are suited to non-blocking IO with subprocesses and sockets. However, blocking IO and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers.

Any program written using threads or processes can be rewritten or instead written using coroutines if we so desire.

A coroutine is even more lightweight than a process. Processes, like threads, are created and managed by the underlying operating system and are represented by a multiprocessing.Process object.

This means that coroutines are significantly faster than a process to create and start and take up much less memory. A coroutine is just a special function, whereas a Process is an instance of the interpreter that has at least one thread.

As such, we may have thousands of threads in a Python program, but we could easily have tens or hundreds of thousands of coroutines all in one thread.

When Not to Use Asyncio

There are many misconceptions about Python concurrency and especially around asyncio. Using asyncio does not magically solve all the issues with Python.

For example:

Asyncio works around the Global Interpreter Lock (GIL).
Asyncio is faster than using threads.
Asyncio avoids the need for locks and other synchronization methods.
Asyncio is easier to use than threads.

These are all False.

Only a single coroutine can run at a time by design, and they cooperate to execute. This is just like threads under the GIL. However, the GIL still applies to asyncio code, and the performance of asyncio programs may be affected by the GIL in the same way as multithreaded programs. So asyncio is not faster and is also not multithreaded and not parallel.

Any program you can write with asyncio, you can also write with threads, and it could actually be as fast or even faster. It could also probably be simpler and easier to read and interpret by other developers.

If you need to bypass the GIL and take full advantage of multiple CPU cores in your Python program, you may want to consider using a different concurrency framework like multiprocessing and other libraries such as concurrent.futures. These frameworks allow you to create multiple processes, each with its own Python interpreter and GIL, which can run in parallel on separate CPU cores.

Any concurrency pitfalls you might expect with threads, you can also encounter with coroutines. You must ensure coroutines are thread safe and safe from deadlocks and race conditions, just like with threads or processes.

Another reason not to use asyncio is that you may not like asynchronous programming. Even though asynchronous programming has been popular for some time now in various programming communities, it is different from procedural, object-oriented, and functional programming, and some developers just don’t like it.

If you don’t like it, don’t use it. It’s a fair reason.

You can achieve concurrency in many ways other than using asynchronous programming, using threads or processes as needed.

You can learn all about threads and processes in our guides:

Running multiple processes in Python - a multiprocessing guide

Best Practices for Asynchronous Programming in Python

These are some tips that will have you writing efficient and maintainable asynchronous code and save you lots of headaches:

Use asyncio's high-level APIs whenever possible

The asyncio library provides several high-level APIs that make it easy to write concurrent code. For example, you can use the gather function to run multiple tasks concurrently or the as_completed function to iterate over a group of tasks as they complete. These APIs can help you write efficient and maintainable code, as they abstract away many of the low-level details of concurrent programming.

Avoid blocking the event loop

The event loop is the heart of asyncio, and it is responsible for scheduling and executing asynchronous tasks. If you block the event loop, you can cause performance issues and disrupt the execution of other tasks. To avoid blocking the event loop, use non-blocking IO operations whenever possible, and avoid using blocking functions such as time.sleep or threading.Lock.

Use async context managers and async iterators

Async context managers and async iterators can help you write asynchronous code that is easy to understand and maintain. For example, you can use async context managers to manage resources that need to be acquired and released asynchronously, or you can use async iterators to iterate over asynchronous data streams in a natural and intuitive way.

Common pitfalls to avoid when doing asynchronous programming

Forgetting to await asynchronous tasks: When you await an asynchronous task, you are telling the event loop to execute that task and pause the current task until the awaited task completes. If you forget to await a task, it will be scheduled to run, but the event loop will not wait for it to complete before moving on to the next task. This can lead to unexpected behavior and race conditions.
Not using asynchronous functions when appropriate: On the other hand, it is important to use asynchronous functions when appropriate, as this can significantly improve the performance and scalability of your application. If you have long-running tasks or tasks that perform IO operations, consider using asynchronous functions to ensure that they do not block the event loop.
Not using synchronization techniques: Any concurrency pitfalls you might expect with threads, you can also encounter with coroutines. While only one coroutine can run within the event loop at one time, they can be suspended and resumed while using a shared variable. You must ensure coroutines are safe from deadlocks and race conditions, just like with threads or processes.

If you keep these practices in mind and avoid the multiple pitfalls related to concurrency and asynchronous programming, you should be able to develop Python programs that can process many tasks concurrently.

You may even want to experiment with running asynchronous tasks combined with other concurrency techniques, especially when pairing asyncio with multiprocessing and concurrent.futures.Executor instances.

References

How are you using asyncio in your programs?

I’d love to hear about it. Please let me know in the comments.

Do you have any questions?

Leave your question in a comment below, and I'll answer it with my best advice.

Why I Started the Atomic-fns Library

HackerCulture — Sat, 05 Nov 2022 15:00:11 +0000

Open source is more important than ever, and more companies are realizing the benefits and increasing their adoption. As long as it's something popular that can speed up the development process, why not use it? It doesn't cost anything!

Well, it doesn't cost money... But I've found there are more subtle costs that could end up having a huge impact on the business, and we should consider them before introducing any dependencies to existing products or processes.

There's the cost of distributing the libraries you're using (and their dependencies) to your customers in your builds. That means your customers need to download a larger bundle, which makes their experience a bit slower.

There's the maintenance cost of outdated libraries, which often include outdated dependencies. Those could have known vulnerabilities with known exploits (and fixes), which can damage the brand, business, and customers.

When a library is no longer maintained, it's almost impossible to get support or request new features. The issues pile up on an endless list of open bugs. And those bugs are almost never fixed when you need them to be. You can count on Murphy's Law to kick in here.

Then there's the cost of the workarounds. Once you start uncovering bugs and you realize nobody is coming to save you, there are two possible outcomes: you either stop using that library, or you try to make it work on your own, taking on weeks of additional work.

On the other hand, there are specific characteristics that separate the quality open-source projects from their abandoned counterparts:

Bundle sizes. Good projects are modular, so you can import only what you need and keep your bundle sizes low. JavaScript libraries can be checked using bundlephobia.com – also, look for libraries that are tree-shakeable.
Source code. If you aren't able to inspect the code, don't use it because it could explode. But also, is the code clear? Are there comments? Could you explain how it works to others?
Up to date. Look at the commit history. Are there recent commits and bug fixes landed? Are they keeping their dependencies up to date?
Test coverage. If there are no tests, how do we know the code does what it's supposed to do? Browse through the tests and make sure that they cover enough cases.
Open issues. Check the project's open issues to make sure the maintainers are active and listening. Check for a roadmap too.
No vulnerabilities. Run npm audit after adding a new dependency to check that it doesn't introduce vulnerabilities. Bonus points if the library you plan to use has zero dependencies.
Documentation. Can you find available docs that show how the library is meant to be used? Official or otherwise, having something you can refer to and share with other team members is super valuable.

Recently, I've had to estimate these costs when deciding on which libraries to use for a new project and was considering using the popular lodash library. Until I started looking at their long list of open bugs, realizing this project has been dead for the past 6 years, and then I read their source code, ridden with custom macros and over-engineered abstractions (see clone). And then it hit me...

This must be why there are so many open bugs. It takes so much effort to understand just one basic function that most people won't even try. Hell, even the original maintainers probably can't understand it at this point because of all the abstractions.

So I started working on a similar library to replace my lodash imports, and a handful of other libraries. But the main goal is that it should be easy to learn, easy to test, and easy to maintain. There are plenty of docs available to get you started, and I hope you join the conversation on github to help shape its future.

[

GitHub - adrian2x/atomic-fns: Like Lodash, but for ESNext and with types.

Like Lodash, but for ESNext and with types. Contribute to adrian2x/atomic-fns development by creating an account on GitHub.

GitHubadrian2x

](https://github.com/adrian2x/atomic-fns)

A Gentle Introduction to Python Programming

HackerCulture — Fri, 28 Jan 2022 14:00:00 +0000

Programming offers a chance to develop creativity, reasoning, and problem-solving skills. As a programmer, you get to bring ideas to life through code and using logic to create programs that can be executed by a computer.

If you know how to write computer programs, you can do all sorts of useful things. You might not be able to write programs to control cars, traffic lights, or your fridge right away, but you could make web pages, your own games, or even a program to help you with your homework.

When faced with obstacles, programming requires you to use problem-solving to find solutions. While it can be challenging (and at times frustrating), programming is also a fun and rewarding activity! The skills gained from programming can be beneficial in multiple professions, even if your chosen career is not directly related to computers.

How to Learn to Code

Like anything you try for the first time, it’s always best to start with the basics. Gymnasts don't usually do backflips on their first try. If you jump ahead too quickly, not only will the basic ideas not stick in your head, but you’ll also find the content of the later articles more complicated than it actually is.

Remember that the better you understand the basics, the easier it will be to understand more complicated ideas later on. When you find something frustrating or too challenging, here are some things that I find helpful:

Break a problem down into smaller pieces. Try to understand what a small piece of code is doing or think about only a small part of a difficult idea (focus on a small piece of code rather than trying to understand the whole thing at once).
If that still doesn’t help, it might be best to leave it alone for a while. Sleep on it and come back to it another day. This is a good way to solve many problems, and it can be particularly helpful for computer programmers.

What is a Computer Program

A program is a list of steps for a computer to follow. Programs are made up of lines of code. Each line tells the computer something specific about how to follow these instructions.

Software is a collection of computer programs.

Programs are like thoughts. If you didn’t have thoughts, you would probably just sit on the couch all day, staring at the ceiling. Your thought “get up off the couch” is a command or instruction that tells your body to stand up. In the same way, computer programs use commands to tell computers what to do.

The Python Programming Language

Like humans, computers use multiple languages to communicate—these are called programming languages. A programming language is simply a way to talk to a computer by using instructions that both humans and computers can understand.

The Python programming language has many features that make it extremely useful for beginners. Most importantly, you can use Python to write simple, efficient programs quite quickly. Python doesn’t use as many complicated symbols as other programming languages, which makes it easier to read and a lot friendlier for beginners.

The Python programming language was actually named after the Monty Python’s Flying Circus TV show, not after the snake.

Installing Python

To install Python, navigate to python.org/downloads and click on Download Python. At the time of this writing, the latest version is 3.11.1. Make sure you download a version higher than 3.10.

Once downloaded, double-click the file to run the installer.
Make sure you select Add Python to PATH, and accept all the defaults as you click through the installer.

When the setup is complete, you will find your Python folder inside the Applications folder on your computer.

Now that you have Python installed, we can write our first program!

For Windows

Search for the Command application in the Start Menu or the Windows search box.

Enter python in all lowercase. You should see a message from Python like this:

$ python
Python 3.11.0 (main, Jan  9 2023, 15:58:34) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Here, you can play around with the Python interpreter. Note the three greater-than signs (>>>) are called the prompt. This means Python is waiting for our input.

For Mac

Python is already installed on most Macs, but it's probably an outdated version and not the one you want.

To see which version of Python is installed, open the Terminal application from the Applications folder and enter python3. You should then see the Python version and prompt.

Note: A shell window (or terminal) is a command-line interface on a computer where you can type commands to the operating system and receive text output in return.

Enter the following line into the prompt:

print("Hello World")

You should see the "Hello World" line printed out.

To close the terminal session, press Ctrl-Z and Enter, or type exit() and press Enter.

But wait a second... you may be thinking, "How is this useful?". That's a totally fair question. Let's look at a more realistic example.

Copy and paste the following code into your Python shell, and press Enter.

You should now see today's date and time printed out!

Note: In Python, the hash symbol (#) indicates a comment. Anything following a hash char in your code is ignored by Python; they are just there for human readers. You can use comments to write notes in plain English within your programs.

Comments can also be used to experiment with your code. If you want to temporarily "turn off" some code so that it doesn't run but don't want to delete it entirely, start the line with a #. This will convert the line into a comment, which Python will ignore when running the program.

Let's take a look at the code starting with this line:

from datetime import datetime

This imports a module called datetime from somewhere, which we will use to display the current date and time. For now, you probably have no idea where it's coming from, and that's completely okay.

The second line is a bit different:

today = datetime.now()

It kind of sounds like this line is saying: today is equal to a datetime now, maybe? Or more like today is datetime now.

That's pretty much exactly what this line does:

It creates a new variable name called "today".
It assigns the result of datetime.now() to the variable today.

Finally, this prints out the value of the variable today.

Now, try to change the name today in the last line to something else, like now, and run the code again.

What to do if you see an error

You may see an error this time, and the output is just the Python interpreter telling us where the problem is.

Traceback (most recent call last):
  File "/var/shared-data/code/main.py", line 6, in <module>
    print(now)
          ^^^
NameError: name 'now' is not defined. Did you mean: 'pow'?

A NameError happens when we either forget to set a variable's name before using it or we made a spelling error when entering the variable name.

The output also reports that this error happened in line 6 of the file main.py. This can help us spot the error quickly.

If we change both of these lines to use the same name "now", the code works again. Try copying these lines into your shell now:

now = datetime.now()
print(now)

Many programming mistakes are simple typos of one character in one line of code. If you spend a long time trying to find one of these mistakes, you're not alone. Many skilled and experienced programmers spend hours looking for these types of small mistakes. Try to laugh about it and move on, knowing that this will happen a lot as you learn to code.

Variables and Operations

Variables are a way of storing things in a program. They're essentially a way to name a value, and they point to the given value in memory. Their value can later be retrieved and used in expressions.

To create a variable, we simply type the name on the left side of an equal sign (=). The right side must be an expression. This can be a primitive value or a more complicated expression.

For example, we can assign the result of a calculation to a variable:

weeks = 52
days = 5 * 52
vacation = 5 * 2 + 4
workdays = days - vacation
workweeks = workdays / 7
print(workdays)

To find the value of a variable, we can use print followed by the name of the variable in parentheses. Try copying the code above to your shell, and then press Enter.

We can also tell Python to change the variable so that it contains a new value and also use more than one name or variable for the same value:

days = 7
week = days
print(week)

Here we're changing the value of the variable days to 7 and then, we're saying that we want the variable week to refer to the current value of the variable days.

However, week isn't a very useful name for a variable because it doesn't tell us much about what the variable is used for. Let's call our variable days_in_week instead.

days_in_week = 7
print(days_in_week)

This makes it clear that we're talking about the number of days in a week.

Variable names can be made up of letters, numbers, and underscores (_), but they can't start with a number.

You can use any lowercase letters for variable names, but the name cannot contain a space, so we use underscores to separate words.

You should not use uppercase letters in variable names because they have a special meaning that we'll discuss another time. Python is a case-sensitive language, which means that today is not the same as Today.

The order of operations

You can do all the basic math operations in Python using the corresponding operators, as you saw in that example.

We use parentheses in programming languages to control the order of operations. An operation is anything that uses an operator. Multiplication and division have a higher order than addition and subtraction, so they’re performed first. In other words, if you enter an equation in Python, multiplication or division is performed before addition or subtraction.

For example, in this expression, the numbers 2 and 10 are multiplied first, and then the number 5 is added to the product:

5 + 2 * 10

The result is 25.

We can change the order of operations (and the result) by adding the parentheses around the numbers:

(5 + 2) * 10

The result of this expression is now 70 because the parentheses tell Python to add 5 to 2, and then multiply the result by 10.

Parentheses can be nested, which means that there can be parentheses inside parentheses, like this:

((5 + 2) * 10) / 7

In this case, Python evaluates the innermost parentheses first, then the outer ones, and then the final division. So it adds 5 to 2, then multiplies the result by 10, and then divides that by 7.

Saving Your Python Programs

As we saw, we can write Python code directly into the shell window... but Python programs wouldn’t be very useful if you needed to rewrite them every time you wanted to use them.

Sure, it might be fine for short programs, but a large program could have millions of lines of code!

Luckily, we can save our programs for future use.
To create and save a new program, enter idle in the Windows search box, and select IDLE (Python 3.11.1 64-bit). In a Mac, open your Applications folder and open the Python 3.11 folder to find IDLE.

IDLE (Integrated DeveLopment Environment) is a Python file editor which let you write programs in Python.

When you open IDLE, you will see the Python shell. Click File › New Window. An empty window will appear.

Paste the code from the previous example into the window, then click File ▸ Save. Enter hello.py as the file name, and save it to your desktop. Then, you can click Run › Run Module to run it.

An alternative to IDLE for Python is Pycharm, which is loved by many Python programmers. You can download the free community Pycharm from their website.

We prefer using Pycharm because it provides a more user friendly environment. It really works like a charm! You can use it to create new programs and to run saved programs directly from within Pycharm.

Run your Python program from the Terminal

We can also run our program from the Terminal by taking these steps:

For Windows

Open the Command Prompt from the Start Menu or Windows search.
Enter cd Desktop in the command window to change the directory to the folder containing our program.
Type python hello.py to run the program.

For Mac and Linux

Open the Terminal app from Applications.
Enter cd ~/Desktop in your terminal to change the directory to the folder containing the program.
Type python3 hello.py to run it.

With some luck, you should see the result of your Python program printed out!

Recap

You just learned how to set up Python and how to start playing with it! Also:

How to save your programs
How to create variables to name values
How to use parentheses to control the order of operations.

Now would be a great time for a celebratory dance! 🎉

Setting up Python and virtualenv with pyenv

HackerCulture — Sat, 15 May 2021 19:00:00 +0000

In this one, I’ll show you how to set up your local machine for python development so you can get started with python today. Let’s go!

Python comes preinstalled in most operating systems by default. But sometimes you may want to upgrade to a newer version of Python or a new version of a library that you depend on which is not compatible with your current version of Python?

If you upgrade your system version, you may introduce changes that are not backward compatible, and you may find yourself hours deep into the rabbit hole of upgrading your entire system.

What we should do instead is use a library like Pyenv for installing multiple versions of Python and switching between them as needed.

We'll also see how virtual environments can help us isolate changes between projects and how to keep project dependencies pinned to a specific version of Python.

Requirements

Ubuntu 20.x / WSL
Internet connection

Open up your terminal (if you’re using WSL, open up your Ubuntu terminal). First, we will update our package index and upgrade any installed packages.

sudo apt update && sudo apt upgrade -y

Enter your password if prompted and wait for the terminal to finish.

…

Now, we will install all the python toolchain dependencies and build tools. Even though Ubuntu ships with python installed, we still need some other libraries to build and run python packages. This is the recommended list of dependencies for sane build environments, according to pyenv instructions.

sudo apt install -y make build-essential build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev python-dev python3-dev python3-pip python3-virtualenv python3-venv git

Once that finishes, you should have all the necessary libraries installed. Now for the python specific part.

Pyenv

The recommended way to manage your python dependencies and packages is to encapsulate your builds inside a python virtual environment. You might be wondering if you really need this, the answer (unless you want to spend most of your time in package versioning hell) is definitely yes.

We’ll use pyenv to manage python installations.

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

We also need to add these lines to your .bashrc (or .zshrc if you use zshell )

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Finally, you will need to reload your terminal session to apply the changes, or run source ~/.bashrc.

Now, let’s go ahead and make sure we have Python installed.

python3 --version
# Python 3.8.6

So my current version is 3.8.5 but there are newer versions out there. We can install them using pyenv without messing with the default system version.

pyenv install 3.11

Once that’s done, we can list all the installed versions with:

pyenv versions
* system (...)
  2.7.18
  3.8.13

You should see some output like that, which tells me I have a system version, and others installed and available.

The reason why this matters is that some legacy python code needs to run with python2 while newer code should be using python3 versions. Also, when a new version of python comes out, you can try it in a new project by installing it with pyenv without having to worry about other legacy code not being able to run on the newer version.

Python virtual environments

Let’s introduce virtual environments by looking at a problem you might encounter. You start working on some project which uses some pip package, like pandas. Without knowing about virtualenvs, you just do what one does to install pandas like pip install pandas and just continue your work.

All is good and well… except for the fact that now you installed a version of pandas in your system dependencies, which means some code you wrote months ago that depended on pandas too is now broken because pandas changed versions and now it’s incompatible with your old code. That can be very messy.

What we really want is to make sure whatever version of pandas we build our project with never changes behind the scenes , so our code stays working even though pandas itself might change in the future. This is known as locking your dependencies , and that’s exactly what virtualenvs are for.

Let’s install virtual environments on top of pyenv.

git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv

echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.profile

Now I’ll show you how you can use virtual environments to isolate dependencies for a new project. I recommend creating a src directory in your root dir and putting all your projects in there. Like this

exec "$SHELL"

mkdir ~/src/new-project
cd ~/src/new-project

pyenv virtualenv 3.11 venv

This just creates a new virtualenv which will use python3.10-dev called venv. You can use any version of python installed by pyenv and any name you like, I just happen to use venv for consistency.

The last step is to actually activate the environment, so every command we run in this directory is isolated from the rest of our python installations.

pyenv activate venv

Now, you should see something like this

(venv) user@epyc02:~/src/new-project$

That first part on your path is telling us we are inside a virtual environment called venv.

BTW you can exit the virtual environment with pyenv deactivate, in case you want to switch between projects or install some packages outside your project.

Wrapping up

OK so now you are all set with python and you can install any dependencies in your project by running pip install <package>. This will save your packages inside of your venv directory and they will be local to your project. If you’re using git it’s recommended to ignore everything under your venv directory, so go ahead and add venv to your gitignore. Note: if you called it something other than venv, substitute your environment’s name below instead of venv.

echo 'venv' >> .gitignore

That way if you deploy your code later or someone is pulling your code but has a different type of CPU when they install the packages they will actually get the right build for their particular environment.

And that is all for our python environment setup. Go write some code!

DEV Community: HackerCulture

The Ultimate Python Programmer Roadmap (2023)

Table of Contents

Python Basics

Intermediate Python

Advanced Python Topics

Python for Web Development

Python for Automation

Python for DevOps

Python for Data Science

Python for Machine Learning

Tips for learning Python programming

Python asyncio - A Guide to Asynchronous Programming

Table of Contents

Introduction to Asynchronous Programming

Getting Started with Asyncio in Python

Understanding the async, await syntax

What is the Event Loop

Creating and customizing event loops

What Are Coroutines

What's so great about coroutines

How to Run Blocking Tasks with Asyncio

Using Asyncio for Network Programming

HTTP server

WebSocket server

WebSocket client

Advanced Asyncio Concepts

How to use asyncio.gather() to wait for multiple async tasks

How to use asyncio.wait() to manage multiple asynchronous tasks

Chaining multiple asynchronous tasks

When to Use Asyncio

When Not to Use Asyncio

Best Practices for Asynchronous Programming in Python

Use asyncio's high-level APIs whenever possible

Avoid blocking the event loop

Use async context managers and async iterators

Common pitfalls to avoid when doing asynchronous programming

References

Why I Started the Atomic-fns Library

A Gentle Introduction to Python Programming

How to Learn to Code

What is a Computer Program

The Python Programming Language

Installing Python

For Windows

For Mac

What to do if you see an error

Variables and Operations

The order of operations

Saving Your Python Programs

Run your Python program from the Terminal

For Windows

For Mac and Linux

Recap

Setting up Python and virtualenv with pyenv

Requirements

Pyenv

Python virtual environments

Wrapping up