DEV Community: Bastien ANTOINE

Debugging Python programs without an IDE

Bastien ANTOINE — Wed, 29 Jun 2022 21:39:04 +0000

Note: this article was originally published on my personal blog. Check it out here.

When writing a program, making sure it works the way it's supposed to is a crucial part of the process. This means you have to test your code, and you almost always end up with bugs that you have to debug.

When you are using a IDE, you often have nice tools to do that, with the ability to place breakpoints, observe the call stack, the content of the locals... But sometimes you find yourself in situation where your program lives somewhere else and you can't use those nice modern tools, and you have to find an other solution.

If you are using Python, have some good news for you: the standard lib comes with a basic debugger included, pdb which we'll discover today.

What is pdb?

pdb, as in Python DeBugger, is an interactive source code debugger for Python programs. It is part of the standard library, which mean that if you have Python, you have it. Just like any debugger it features builtin breakpoints capabilities, context and stack introspection as well as code evaluation.

How to use it?

There are a basically three way to run a program under pdb. The first one is using the run() function inside a REPL¹, the second is by putting a breakpoint inside your code with the set_trace() function. The last one will also set a breakpoint, but this time using the builtin breakpoint() function.

Assume you have the following piece of code you want to debug:

# mymodule.py
from math import sqrt

def compute(a: int, b: int) -> float:
    c = a ** 2 + b ** 2
    c = sqrt(c)
    return c

def launch():
    result = compute(3, 4)
    print(result)

if __name__ == '__main__':
    launch()

Using `run()` inside a REPL

The first way to use pdb is bt using the run() function inside a REPL¹:

>>> from mymodule import launch
>>> import pdb
>>> pdb.run("launch()")
> <string>(1)<module>()
(Pdb)

After providing the name of the function to run to the run() function, we find ourselves with a different shell, as we can see with the (Pdb). That's the pdb shell, where we can explore our code, set breakpoint, show the stack... We'll talk about how to interact with it later.

Setting a breakpoint with `set_trace()`

An other way is by putting a breakpoint inside your code with the set_trace() function. When you put it inside your code, Python will stop the current execution each time it reaches it, and will launch a pdb shell for you to interact. Now assume I put a breakpoint inside the previous code at the beggining of the launch() function:

  from math import sqrt
+ import pdb

  def compute(a: int, b: int) -> float:
      c = a ** 2 + b ** 2
      c = sqrt(c)
      return c

  def launch():
+     pdb.set_trace()
      result = compute(3, 4)
      print(result)

  if __name__ == '__main__':
      launch()

If I launch this updated script with the regular python (and not inside a REPL):

> python mymodule.py
> /home/user/mymodule.py(11)launch()
-> result = compute(3, 4)
(Pdb)

We end up with basically the same input, where we can interact with pdb.

Using the builtin `breakpoint()` function

There's a third way to run pdb. It is quite similar to the previous one, but this time we'll be using a builtin function: breakpoint(). This function exists since Python 3.7, and acts as some kind of proxy to your debugger, allowing you to switch between various Python debugger, without having to update your code.

Its default behavior is to call pdb.set_trace() when called, which is what matters for us here:

  from math import sqrt
- import pdb

  def compute(a: int, b: int) -> float:
      c = a ** 2 + b ** 2
      c = sqrt(c)
      return c

  def launch():
-     pdb.set_trace()
+     breakpoint()
      result = compute(3, 4)
      print(result)

  if __name__ == '__main__':
      launch()

If I launch this updated script with the regular python (and not inside a REPL):

> python mymodule.py
> /home/user/mymodule.py(11)launch()
-> result = compute(3, 4)
(Pdb)

We can see that we end up with the same output as when we were using pdb.set_trace().

Interacting with pdb

As you may have noticed, all interaction with pdb must be done using the keyboard, as it is the case with everything that happens inside a terminal. As such, there are a few commands builtin that allows you to interact with it. Let's run over the most interesting ones.

THe official doc of the pdb module presents a list of all the available commands.

Note on the syntax

Some of them have a both short and long syntax, which mean you can use either one of them for the same result. They are written like this, with the long form between parenthesis: my(command). For the previous example, my is the short version, while mycommand is the long.

Note: all the excerpts shown below have been reduced for the sake of readibility. If you try them yourself, you might see a bit more verbosity, but the core stays the same.

l(ist)/ll

These two commands are really similar: they both print the source code for the current file:

l(ist) will print 11 lines around the current line being executed (ie. 5 lines before, the line executed and 5 lines after)
ll will print all the source code for the current function

The current line in the current frame is indicated by ->.

(Pdb) ll
  4     def compute(a: int, b: int) -> float:
  5         c = a ** 2 + b ** 2
  6         c = sqrt(c)
  7  ->     return c
(Pdb) list
  2     from pdb import set_trace
  3
  4     def compute(a: int, b: int) -> float:
  5         c = a ** 2 + b ** 2
  6         c = sqrt(c)
  7  ->     return c
  8
  9     def launch():
 10         set_trace()
 11         result = compute(3, 4)
 12         print(result)
(Pdb)

c(ont(inue))

This command will simply continue the executions of the program, until the next breakpoint, the end of the program, or any error raised.

(Pdb) ll
  4     def compute(a: int, b: int) -> float:
  5         c = a ** 2 + b ** 2
  6         c = sqrt(c)
  7  ->     return c
(Pdb) continue
5.0
user@computer ~/ >

q(uit)

This command will simply stop the execution of the program and exits right away.

user@computer ~/ > python mymodule.py
-> result = compute(3, 4)
(Pdb) step
--Call--
-> def compute(a: int, b: int) -> float:
(Pdb) next
-> c = a ** 2 + b ** 2
(Pdb) next
-> c = sqrt(c)
(Pdb) quit
Traceback (most recent call last):
  File "/home/user/mymodule.py", line 15, in <module>
    launch()
  File "/home/user/mymodule.py", line 11, in launch
    result = compute(3, 4)
  File "/home/user/mymodule.py", line 6, in compute
    c = sqrt(c)
  File "/home/user/mymodule.py", line 6, in compute
    c = sqrt(c)
  File "/usr/lib/python3.7/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.7/bdb.py", line 113, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

Warning: Be careful when using this command, especially when you are using handlers that need to be properly closed, like file handlers or database connections. If the code you are debugging doesn't properly close them in case of error, you might end up with side errors because those handlers weren't properly closed.

Check out below the explanation of the s(tep) and n(ext) commands.

p/pp

evaluate and (pretty) prints an expression in the current context

These two commands are really similar. They both evaluate a given expression in the current context and print its value. p will print just like the regular builtin print() function does, while pp will pretty-print using the pprint module.

(Pdb) p f"a={a}, b={b}"
'a=3, b=4'
(Pdb) pp list(range((a+b)**2))
[0,
 1,
 2,
 3,
 4,
 5,
 6,
 7,
 8,
 9,
 10,
 ...,
 40,
 41,
 42,
 43,
 44,
 45,
 46,
 47,
 48]

n(ext)/s(tep)

These two commands are almost the same. They allow you to move forward in you program, one line at a time, by executing the next line.

Their behavior differ when the next step to execute is a call to a function:

n(ext) will simply execute the function
s(tep) will enter the function and stops at the function signature (ie. the def myFunc(*args, **kwargs):), allowing you to go further using n(ext) or s(tep).

(Pdb) ll
  9     def launch():
 10         set_trace()
 11  ->     result = compute(3, 4)
 12         print(result)
(Pdb) step
--Call--
-> def compute(a: int, b: int) -> float:
(Pdb) next
-> a = 3
(Pdb) next
-> b = 4
(Pdb)

`w(here)`

This command prints the stack trace, with the most recent stack at the bottom.

(Pdb) where
  /home/user/mymodule.py(15)<module>()
-> launch()
> /home/user/mymodule.py(11)launch()
-> result = compute(3, 4)
(Pdb)

`u(p)/d(own)`

These two commands allows you to move up and down inside the stack. u(p) will go to an older stack, ie. up in the list, while d(own) will go to a newer one, ie. one at the bottom of the list.

(Pdb) up
-> launch()
(Pdb) list
 10         set_trace()
 11         result = compute(3, 4)
 12         print(result)
 13
 14     if __name__ == '__main__':
 15  ->     launch()
[EOF]
(Pdb) down
-> result = compute()
(Pdb) list
  6         c = sqrt(c)
  7         return c
  8
  9     def launch():
 10         set_trace()
 11  ->     result = compute(3, 4)
 12         print(result)
 13
 14     if __name__ == '__main__':
 15         launch()
[EOF]
(Pdb)

r(eturn)

This command is useful when you are inside a function. It will continue its execution until it raises or returns.

(Pdb) ll
  4     def compute(a: int, b: int) -> float:
  5  ->     c = a ** 2 + b ** 2
  6         c = sqrt(c)
  7         return c
(Pdb) return
--Return--
> /home/user/mymodule.py(7)compute()->5.0
-> return c
(Pdb) ll
  4     def compute(a: int, b: int) -> float:
  5         c = a ** 2 + b ** 2
  6         c = sqrt(c)
  7  ->     return c
(Pdb)

a(rgs)

This command is really useful when you are insed the scope of a function, as it will print the args this function has been called with.

(Pdb) args
a = 3
b = 4

interact

The command interact can be considered as an evolution of the two above. It will enter in a classic REPL, with the locals of the current scope loaded in the globals, meaning you can interact with it, just like in a regular Python console.

(Pdb) interact
*interactive*
>>> from pprint import pprint
>>> pprint(locals())
{'__annotations__': {},
 '__builtins__': <module 'builtins' (built-in)>,
 '__cached__': None,
 '__doc__': None,
 '__file__': '/home/user/mymodule.py',
 '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x107bf5ca0>,
 '__name__': '__main__',
 '__package__': None,
 '__return__': 5.0,
 '__spec__': None,
 'a': 3,
 'b': 4,
 'c': 5.0,
 'compute': <function compute at 0x107c41310>,
 'launch': <function launch at 0x107e0e940>,
 'pprint': <function pprint at 0x1082c6c10>,
 'set_trace': <function set_trace at 0x1082df700>,
 'sqrt': <built-in function sqrt>}
>>> # Ctrl+D to exit and return to the pdb console
now exiting InteractiveConsole...
(Pdb)

`ipdb`, the evolution

Do you know IPython? It is a modern Python console that extends the capabilities of the classic builtin Python shell by offering introspection, tab completion, syntaxing coloring, as well as history. If you don't know it, I can't recommend it enough. More information can be found in its GitHub page.

Based on this evolved console, some people have designed an evolved version of pdb that uses this REPL to bring all the cool features of IPython inside pdb: it is called ipdb.

Once you have installed it using pip, like any regular Python package, you can use like the same way you would use pdb:

- import pdb
+ import ipdb

  def compute(a: int, b: int) -> float:
      c = a ** 2 + b ** 2
      c = sqrt(c)
      return c

  def launch():
-     pdb.set_trace()
+     ipdb.set_trace()
      result = compute(3, 4)
      print(result)

  if __name__ == '__main__':
      launch()

Once run, the commands are the same as with pdb, excpet with IPython formatting applied (which you can't really see with the markdown formatting applied here. You'll have to either trust me or try it by yourself²):

user@computer ~/ > python mymodule.py
     10     set_trace()
---> 11     result = compute(3, 4)
     12     print(result)

ipdb> ll
      9 def launch():
     10     set_trace()
---> 11     result = compute(3, 4)
     12     print(result)
     13

ipdb> step
--Call--
      3
----> 4 def compute(a: int, b: int) -> float:
      5     c = a ** 2 + b ** 2

ipdb> next
      4 def compute(a: int, b: int) -> float:
----> 5     c = a ** 2 + b ** 2
      6     c = sqrt(c)

ipdb> args
a = 3
b = 4
ipdb>

Note on running ipdb with `breakpoint()`

As said above, since Python 3.7, there's the builtin command breakpoint() that allows to call a Python debugger using a single command. The default behavior of this command is to run pdb.set_trace() when called. If you are using ipdb, you can change this behavior so that it uses this debugger instead.

To do that you'll have to set the environment variable PYTHONBREAKPOINT to ipdb.set_trace, so that Python will know which debugger to use:

- import ipdb

  def compute(a: int, b: int) -> float:
      c = a ** 2 + b ** 2
      c = sqrt(c)
      return c

  def launch():
-     ipdb.set_trace()
+     breakpoint()
      result = compute(3, 4)
      print(result)

  if __name__ == '__main__':
      launch()

user@computer ~/ > export PYTHONBREAKPOINT=ipdb.set_trace
user@computer ~/ > python mymodule.py
     10     set_trace()
---> 11     result = compute(3, 4)
     12     print(result)

ipdb>

Some drawbacks

Now, like every nice tools, pdb, and its modern version ipdb, have some issues.

An interactive terminal is required

This may sound obvious, but I can assure you sometimes you forget about this and are reminded the hard way when you are trying to debug a application that's inside a Docker container.

To be a bit more precise, to be able to properly use (i)pdb, you need to have access to a terminal in which you have access to both its input and output file descriptors, so that you can send commands to (i)pdb and see the output of the debugger.

This means that if you are trying to debug an application that's running inside a container, a VM, or even a Kubernetes pod, and where you don't have any acces further than the logs, you'll have to find another solution.

In this case, there are some pdb extensions that allows you to connect to the debugger using telnet, but that will the subject of a next article.

The interface is not very user-friendly

As you may have noticed, the interface isn't really user-friendly, if we can even call it a real user interface. You have to remember where you are in your code, you can quickly see the content of your current context, setting new breakpoints within (i)pdb is kinda hard...

This may be enough for some small applications, but once you start to debug more complex programs, this might become hard to use, if not impossible. In this you may find some more advanced tools more useful, like pudb. pudb is a console-based visual debugger for Python that is absolutely awesome, but I'll save the presentation for a next article.

Sources

the REPL, or Read-Eval-Print Loop, is the fancy name of the console, or shell. Its name comes from its design as it is a loop where each iteration consists of the same three steps: ↩
1. *read*ing of the user input
2. *eval*uation of the input
3. *print*ing of the result
You definitely should! ↩

Discovering Python Namespace Packages

Bastien ANTOINE — Sun, 09 Jan 2022 15:02:18 +0000

Note: this article was originally published on my personal blog. Check it out here.

In a recent post about the __path__ attribute, I explained that it would be possible to manipulate a library's path to dynamically extend its functionalities. At the end of my article I detailed that this possibility has already been thought of was integrated in the import system since Python 3.3 with PEP 420, under the name of namespace package.

Today I'd like to go a bit more in details about this.

What is a package in Python?

The Python glossary defines a package as:

A Python module which can contain submodules or recursively, subpackages. Technically, a package is a Python module with a __path__ attribute.

So a package in Python is basically a set of .py files (modules) organized in subdirectories (subpackages).

The __path__ attribute is specific to the packages. It is used by the import system when trying to import modules of subpackages. Basically its function is the same as sys.path, but limited to the scope of a package. I'll let you check out my article on the __path__ attribute about it for more information.

The specificity of the namespace packages

A namespace package is a special kind of Python package: among other differences with the regular packages, the most important and notable difference is that its source files can be splited across multiple locations.

This means that the following structure can be considered a namespace package, named parent (I'll explain later on how this can be used):

/home/myUser/dev/
└── package/
    └── parent/
        └── child/
            └── foo.py
/var/usr/lib
└── package/
    └── parent/
        └── child/
            └── bar.py

Now with the following example, I would have two distinct packages, both named parent, one in /home/myUser/dev/ and another in /var/usr/lib¹.

/home/myUser/dev/
└── package/
    └── parent/
        ├── __init__.py
        └── child/
            ├── __init__.py
            └── foo.py
/var/usr/lib
└── package/
    └── parent/
        ├── __init__.py
        └── child/
            ├── __init__.py
            └── bar.py

Differences with the regular packages

Like I already mentioned, the most notable difference between regular and namespace packages is that the latter can be splitted across multiple locations.

This ability to split the sources also impacts the package's __path__ attribute: instead of being a list with a single element being the path of the package's __init__.py, it is an iterable of paths, containing all the locations of the package's sources.

Keen eyes might have noticed also that I ommitted the __init__.py files in the structure of the namespace package example above. That's not a mistake, actually namespace package are recognized and detected by the Python import system by their lack of __init__.py files.

Step by step basic example

Take the following structure as example:

/home/myUser/dev/
└── packageExtension/
    └── parent/
        └── child/
            └── bar.py
/var/usr/lib
└── mainPackage/
    └── parent/
        └── child/
            └── foo.py

To make it work as a namespace package, we first need to make sure that both sources can be found in sys.path. For the sake of the examples we'll add them by hand in it:

>>> import sys
>>> sys.path.extend([
  '/var/usr/lib/mainPackage',
  '/home/myUser/dev/packageExtension',
])

Now we can try to load the parent package, and see where Python finds its source files:

>>> # Continue from previous stage
>>> import parent
>>> parent.__path__
_NamespacePath([
  '/var/usr/lib/mainPackage/parent',
  '/home/myUser/dev/packageExtension/parent'
])

What's interesting here is the value of parent.__path__: like said before, it's an iterable of strings for all the locations in sys.path where a namespace package named parent has been found.

Now if we try to import the child package, we should have the same thing:

>>> # Continue from previous stage
>>> import parent.child
>>> parent.child.__path__
_NamespacePath([
  '/var/usr/lib/mainPackage/parent/child',
  '/home/myUser/dev/packageExtension/parent/child'
])

And now we can import both foo and bar modules under the same package name, even though they are in different locations:

>>> # Continue from previous stage
>>> import parent.child.foo
>>> parent.child.foo.__file__
'/var/usr/lib/mainPackage/parent/child/foo.py'
>>> import parent.child.bar
>>> parent.child.bar.__file__
'/home/myUser/dev/packageExtension/parent/child/bar.py'

Dynamic loading of package parts

Now we've seen that to be able to load differents portions of a namespace package, first they all need to be accessible under sys.path. What's cool is that if during the run, a new portion of a namespace package that's already imported were to be added to sys.path, it'll be able to be loaded under the same namespace package.

Take the following structure as example. It the same as previously, but with /home/myOtherUser/dev/otherPackageExtension/ added:

/home/myUser/dev/
└── packageExtension/
    └── parent/
        └── child/
            └── bar.py
/home/myOtherUser/dev/
└── otherPackageExtension/
    └── parent/
        └── child/
            └── baz.py
/var/usr/lib
└── mainPackage/
    └── parent/
        └── child/
            └── foo.py

Now if we continue back where we left the previous example:

>>> # Continue from previous stage
>>> parent.__path__
_NamespacePath([
  '/var/usr/lib/mainPackage/parent',
  '/home/myUser/dev/packageExtension/parent'
])
>>> import parent.child.baz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'parent.child.baz'

Now if we add /home/myOtherUser/dev/otherPackageExtension/ to sys.path, the namespace package parent's __path__ should be recomputed to include the newly found portion:

>>> # Continue from previous stage
>>> sys.path.append('/home/myOtherUser/dev/otherPackageExtension/')
>>> sys.path
[
  ...,
  '/var/usr/lib/mainPackage',
  '/home/myUser/dev/packageExtension',
  '/home/myOtherUser/dev/otherPackageExtension'
  ...,
]
>>> parent.__path__
_NamespacePath([
  '/var/usr/lib/mainPackage/parent',
  '/home/myUser/dev/packageExtension/parent',
  '/home/myOtherUser/dev/otherPackageExtension/parent'
])

Which mean we can now import parent.child.baz:

>>> # Continue from previous stage
>>> import parent.child.baz
>>> parent.child.baz.__file__
'/home/myUser/dev/otherPackageExtension/parent/child/baz.py'

Limitations

There are some limitations to this :

All portions of a namespace package must be declared as namespace packages (ie. without __init__.py files). So the following is a namespace package with 2 portions (one in /home/myUser/dev/packageExtension and one in /var/usr/lib/mainPackage):

/home/myUser/dev/
└── packageExtension/
    └── parent/
        └── child/
            └── bar.py
/var/usr/lib
└── mainPackage/
    └── parent/
        └── child/
            └── foo.py

But the following is not (note the __init__.py inside /home/myUser/dev/packageExtension), even though /var/usr/lib/mainPackage is a namespace package:

/home/myUser/dev/
└── packageExtension/
    ├── __init__.py
    └── parent/
        └── child/
            └── bar.py
/var/usr/lib
└── mainPackage/
    └── parent/
        └── child/
            └── foo.py

All the portions of a namespace package must be foundable in sys.path, though they don't have to be already in it when loading the namespace package, the __path__ is dynamically computed.

This point has an implication on the performances: because the path is recomputed each time a portion is loaded, the importation of a module of package can take a bit more time with a namespace package, than with a regular one.

Possible usage

Now that we've discussed a bit the theory and seen a few examples, let's try to figure out real world applications. I've found two categories of usage of namespace applications:

A unique interface for different portions of packages provided by various vendors.

Imagine a low-level package that allows you to access various informations on your system, both software and hardware. You could have something that look a bit like this, where each part is provided by the manufacturer of the part at stake:

gpu_vendor/
└── system_management/
    └── hardware/
        └── gpu.py
cpu_vendor/
└── system_management/
    └── hardware/
        └── cpu.py
ram_vendor/
└── system_management/
    └── hardware/
        └── ram.py
os_vendor/
└── system_management/
    └── software/
        └── os.py
...

With this structure, and assuming all the portions are accessible within sys.path, you'd have a single namespace package system_management under which you could import system_management.hardware.gpu or system_management.software.os.

Adding optional extensions to a package

In this example you would have a main namespace package that provides basic functionalities which you could extend at will, by installing other portions. Have a look at the next section for a more concrete example.

More concrete example

Let's take the example of an image manipulation library. It has a base library, and additional extensions that allows the manipulation of various type of images (jpg, png,...). It has the following structure:

example/
├── baseLib/
│   └── img_lib/
│       └── lib/
│           ├── base.py
│           └── settings.py
├── jpg/
│   └── img_lib/
│       └── lib/
│           └── jpg.py
├── gif/
│   └── img_lib/
│       └── lib/
│           └── gif.py
├── png/
│   └── img_lib/
│       └── lib/
│           └── png.py
└── svg/
    └── img_lib/
        └── lib/
            └── svg.py

Each image module simply declares its mimetype with the constant MIMETYPE.

The base module reads the list of extensions declared in settings.py, and will load them. After that, it'll be able to show the mimetype of all the extensions available.

>>> import os, sys
>>> # For the sake of the example, add all the portions of the namespace package in sys.path by hand.
>>> os.getcwd()
<...>/example
>>> sys.path.extend([
   ...:     os.path.join(os.getcwd(), 'baseLib'),
   ...:     os.path.join(os.getcwd(), 'png'),
   ...:     os.path.join(os.getcwd(), 'jpg'),
   ...:     os.path.join(os.getcwd(), 'gif'),
   ...:     os.path.join(os.getcwd(), 'svg'),
   ...: ])
>>> from img_lib.lib import base
>>> base.AVAILABLE_EXTENSIONS
['png', 'jpg']
>>> base.get_loaded_mimetypes()
['image/png', 'image/jpeg']
>>> base.AVAILABLE_EXTENSIONS.extend(['svg', 'gif'])
>>> base.load_modules()
>>> base.get_loaded_mimetypes()
['image/png', 'image/jpeg', 'image/svg+xml', 'image/gif']

All the source code of this example is available in this GitHub repo.

Sources

this setup would prevent you to import both modules foo.py and bar.py at the same time if both /home/myUser/dev/package and /var/usr/lib/package where to be in sys.path at the same time. ↩

Http Requests Cheatsheet

Bastien ANTOINE — Mon, 27 Dec 2021 09:21:09 +0000

Note: this article was originally published on my personal blog. Check it out here.

When working around APIs, whether it is building or playing with, you often end up making API requests over and over that all look alike, and yet there are some stuff that you can't remember. Like how would you provide form data with cURL? Or basic authentication with requests? This article serves as a cheatsheet on all those common stuff.

Selecting the HTTP method

`requests` (Python)

3 possible ways:

Using the requests.request method:

resp = requests.request('GET', 'https://httpbin.org/get')
resp = requests.request('POST', 'https://httpbin.org/post')
resp = requests.request('PUT', 'https://httpbin.org/put')
resp = requests.request('PATCH', 'https://httpbin.org/patch')
resp = requests.request('DELETE', 'https://httpbin.org/delete')
resp = requests.request('HEAD', 'https://httpbin.org/get')

Using the requests.<method> methods:

resp = requests.get('https://httpbin.org/get')
resp = requests.post('https://httpbin.org/post')
resp = requests.put('https://httpbin.org/put')
resp = requests.patch('https://httpbin.org/patch')
resp = requests.delete('https://httpbin.org/delete')
resp = requests.head('https://httpbin.org/get')

Using a prepared request:

req = requests.Request('GET', 'https://httpbin.org/get')
r = req.prepare()
s = requests.Session()
s.send(r)

From requests.PreparedRequest documentation.

cURL

curl -X GET https://httpbin.org/get
curl -X POST https://httpbin.org/post
curl -X PUT https://httpbin.org/put
curl -X PATCH https://httpbin.org/patch
curl -X DELETE https://httpbin.org/delete

If the -X/--request is not provided, cURL will default to GET.

[doc of -X/--request option]

Note on the HEAD requests: to make a proper HEAD request (ie. requesting only the headers), the -X HEAD option will not work, you will need to use -I/--head instead:

> curl -X HEAD https://httpbin.org/get
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 255 bytes remaining to read
> curl -I https://httpbin.org/get
HTTP/2 200
date: Sun, 19 Dec 2021 11:03:31 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

(Tested on cURL 7.64.0 and 7.77.0)

Headers

`requests` (Python)

requests.get(
  'https://httpbin.org/headers',
  headers={
      'X-My-Header': 'Value',
  }
)

[doc of headers parameter]

cURL

curl -H 'X-My-Header: Value' https://httpbin.org/headers

[doc of -H/--headers option]

Authentication

HTTP Basic

`requests` (Python)

2 possible ways:

Using the auth option

requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  auth=('user', 'passwd')
)

Using the headers option

requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': 'Basic dXNlcjpwYXNzd2QK',
  }
)

from base64 import b64encode
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': f'Basic {b64encode('user:passwd')}',
  }
)

Note on using this along with a .netrc file: Authorization headers that are provided this way will be overriden by credentials set in the .netrc file [doc].

As a workaround you can either temporarily comment the authentication part in your .netrc, set the NETRC env var to an empty file, or temporarily set requests.utils.NETRC_FILES = [] [source code] .

cURL

2 possible ways:

Using the -u/--user option

curl -u user:passwd https://httpbin.org/basic-auth/user/passwd

Few things to note when using this option:

cURL will split the username and password on the first colon. This means you can't have one in the username, but you can in the password. If needed, use the solution below with the -H/--header option.
If only the username is provided, cURL will prompt for the password before sending the request

Using the -H/--header option

curl -H 'Authorization: Basic dXNlcjpwYXNzd2QK' https://httpbin.org/basic-auth/user/passwd
curl -H "Authorization: Basic $(openssl base64 <<< user:passwd)" https://httpbin.org/basic-auth/user/passwd

HTTP Bearer

`requests` (Python)

requests.get(
  'https://httpbin.org/bearer',
  headers={
      'Authorization': 'Basic dGhpc0lzQUJlYXJlclRva2VuCg==',
  }
)

cURL

curl -H 'Authorization: Bearer dGhpc0lzQUJlYXJlclRva2VuCg==' https://httpbin.org/bearer

Query parameters

`requests` (Python)

requests.get(
  'https://httpbin.org/anything',
  params={
      'arg1': 'val1',
      'arg2': 'val2',
  }
)

[doc of params parameter]

cURL

No useful options here unfortunately

curl https://httpbin.org/anything?arg1=val1&arg2=val2

Body parameters

`requests` (Python)

There are multiple ways to provide a body, each with their side consequences:

Providing a dict in the data parameter: this will set the Content-Type header to application/x-www-form-urlencoded, unless told otherwise by the headers parameter.

requests.post(
  "https://httpbin.org/post",
  data={'key1': 'value1', 'key2': 'value2'},
)

Providing a raw string in the data parameter: this will not set any value to the Content-Type header.

requests.post(
  "https://httpbin.org/post",
  data='theBodyOfMyRequest',
)

Providing a dict in the json parameter: this will dumps the provided payload as JSON and set the Content-Type header to application/json, unless told otherwise by the headers parameter. The following two request are equivalents:

import json
requests.post(
  "https://httpbin.org/post",
  data=json.dumps({'key1': 'value1 and value2', 'key2': 'value2'}),
  headers={'Content-Type': 'application/json'},
)
requests.post(
  "https://httpbin.org/post",
  json={'key1': 'value1 and value2', 'key2': 'value2'},
)

[doc]

cURL

cURL offers multiple options to send a request with a body, each with their side consequences as well.

Using the -F/--form option

curl -F key1=value1 -F key2=value2 https://httpbin.org/post

Few things to note when using this option:

it forces the method to POST, unless told otherwise with -X/--request option
it sets the Content-Type header to multipart/form-data, unless told otherwise aswell with -H/--header option

Using this option, it is possible to send files or read input from a file by prefixing the filename with @ or <, respectively. This means that if you need to have either a @ or < in the key and/or the value, you need to use the --form-string option.

[doc of -F/--form option]

[doc of --form-string option]

Using -d/--data option

curl -d key1=value1 -d key2=value2 https://httpbin.org/post

Few things to note when using this option:

it forces the method to POST, unless told otherwise with -X/--request option
it sets the Content-Type header to application/x-www-form-urlencoded, unless told otherwise aswell with -H/--header option

Using this option, it is possible to read input from a file by prefixing the filename with @. This means that if you need to have a @ in the key and/or the value you need to use the --data-raw option.

It is also possible to URL-encode the parameter using the --data-urlencode option.

[doc of -d/--data option]

[doc of --data-raw option]

[doc of --data-urlencode option]

Sending files

`requests` (Python)

requests.post(
  'https://httpbin.org/post',
  files={'file': ('index.html', open('index.html', 'rb'), 'text/html', {'Expires': '0'})}
)

[doc]

cURL

curl -F "file=@index.html" https://httpbin.org/post
curl -F "file=@index.html;type=text/html" https://httpbin.org/post

[doc of -F/--form option]

Disabling SSL verification

`requests` (Python)

resp = requests.get(
    'https://httpbin.org/get',
    verify=False,
)

[doc of verify parameter]

cURL

curl -k https://httpbin.org/get

[doc of -k/--insecure option]

3XX redirects

`requests` (Python)

requests default behavior is to follow redirect, unless making a HEAD request (see below for more informations). Use the parameter allow_redirects to disable it:

resp = requests.get(
    'https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302',
    allow_redirects=False,
)

Note on redirects with HEAD requests: when making HEAD requests, requests will not follow 3XX redirects in most case, depending on how the request is made:

How the request is made	Follow 3XX on default
`requests.head(...)`	No
`requests.Session(...).head(...)`	No
`requests.request('HEAD', ...)`	Yes

cURL

cURL's default behavior is to not follow 3XX redirects. To enable this use the -L/--location option:

curl -L https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302

[doc of -L/--location option]

Logging

`requests` (Python)

import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

[source]

cURL

curl -v https://httpbin.org/get

Note that this option only shows requests headers and full response. If you want to see the whole process, including request body, checkout the --trace and --trace-ascii options.

[doc of -v/--verbose option]

Quick intro to Bash I/O redirections, here documents and here strings

Bastien ANTOINE — Mon, 27 Dec 2021 09:20:44 +0000

Note: this article was originally published on my personal blog. Check it out here.

If you have done a bit of scripting in Bash, you may know that you can redirect the input and output of the commands to other commands or files. This article introduces the basic of I/O redirections and explains some of the syntax that may seems cryptic at first.

`stdin`, `stdout` and `stderr`

In Bash every command or script has three files¹ opened when running:

stdin (STanDart INput): the input of the data fed to the program
stdout (STanDart OUTput): the location where the program will output anything it needs to output
stderr (STanDart ERRor): the location where the program will output any error occured during the run

Each of them get assigned to a specific file descriptor (ie. a number assigned to the file used by the OS to handle it): O for stdin, 1 for stdout and 2 for stderr. These numbers allow you to control and use them in your program.

Controlling the program input and output

Controlling the input

Depending on the programm that you want to run, there might be different ways to control on their input is provided: some are built to accept files, some will read only from stdin and others may accept both.

When it comes to directly providing files to the program, I'll let you read their manpage to know how to do so.

When your program is launched directly in a terminal, its stdin is attached to your keyboard, providind you with a way to input the data.

If you want to use something else as the input, there are a few ways:

Using the < operator: this operator allows to use the content of a file as the input data stream.

$ cat >input-file <<EOF
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10
Line 11
EOF
$ grep "^Line 1" <input-file
Line 1
Line 10
Line 11

(See section below about here docs to understand the syntax of the first command.)

You can also use the 0< operator, which is equivalent to <. The 0 here is a reference to the file descriptor 0 attached to stdin, but because a program only has one stdout, it can be abreviated from 0< to <.

Using the pipe | operator: the pipe operator is a special operator that combines input and output redirection to allow chaining commands. See below for more info.
Using here documents and here strings

Here documents and here strings

Here documents (or heredoc) are a special form of input redirection allowing to input a multiline string to a program. The syntax is the following:

COMMAND <<DELIMITOR
...
...
DELIMITOR

DELIMITOR is used to control when to start and stop the input. It can be any string without spaces, and should then be choosed wiselly so that it won't apear in the input:

$ cat <<ThisIsADelimitor
Hello
There
ThisIsADelimitor
Hello
There
$ cat <<1
Hello
There
1
Hello
There

Heredoc allows for parameter expansion as well:

$ export FIRSTNAME='Obi-Wan'
$ export LASTNAME='Kenobi'
$ cat <<EOF
Hello $FIRSTNAME $LASTNAME
EOF
Hello Obi-Wan Kenobi

If you have played a bit with the heredocs, you may have noticed that tabs included in the input were kept:

$ cat <<EOF
        hello
there
EOF
    hello
there

Fortunately you can strip them by prefixing the delimitor with a dash:

$ cat <<-EOF
        hello
there
EOF
hello
there

This will strip all leading tabs, allowing the input to be more readable visually. Note though that this strips only leading tabs, not spaces, so be careful, especially if you copy/paste the example above.

Here strings

Here strings are a more consise version of the here documents. They only allow a single line input, and thus don't need the use of a string delimitor. Like the heredocs, it allows for parameter expansion:

$ cat <<EOF > file
This is
a
multiline
file
EOF
$ export TITLE="Title of the file"
$ export FILE="file"
$ cat - $FILE <<<$TITLE > $FILE.new
$ cat $FILE.new
Title of the file
This is
a
multiline
file

Controlling the output

Like for the input, there are a few ways to redirect the output of a program, but unlike the first, a program has two type of possible outputs, stdout and stderr, so there are ways to control which output gets redirected.

When the program is launched directly in a terminal, its stdout and stderr are attached to the terminal, so everything the program outputs gets printed on your screen.

Controlling `stdout`

Using the > operator: redirects stdout to the provided filename, creates it if needed, otherwise overwrite it.

$ ls -la > list-files.txt

Using the >> operator: same as >, but appends to the end of the file if it already exists

$ ls -lat | head -n3 >> most-recents-files.txt

Controlling `stderr`

Using the 2> operator: same as > for stdout, but for stderr

$ export ERRORFILE=script.errors
$ bad_command1 2>$ERRORFILE
$ cat $ERRORFILE
zsh: command not found: bad_command1

Using the 2>> operator: same as >> for stdout, but for stderr

$ export ERRORFILE=script.errors
$ bad_command1 2>>$ERRORFILE
$ cat $ERRORFILE
zsh: command not found: bad_command1

2>&1: redirects stderr to stdout
&> filename: redirects both stdout and stderr to filename

A note on the pipe operator

The pipe operator | is special operator that allows to chain multiple commands by binding the stdout of one to the stdin of the following:

$ ls -la | wc -l
15

The example above would be roughly the same as the following script:

#!/bin/bash
filename=temp
ls -la >$filename
exec 3<$filename
wc -l <&3
3<&-
rm -rf $filename

Sources

While they may use regular files as input or output these are actually more like data streams, but by convention the UNIX and Linux world treat them like files. ↩

Playing with Python `path` attribute

Bastien ANTOINE — Thu, 23 Dec 2021 17:17:42 +0000

Note: this article was originally published on my personal blog. Check it out here.

Recently, when preparing an article about Python import system, I discovered the __path__ attribute and found some cool stuff to do with it, so I wanted to show what I discovered.

Take a look at the following example:

myPackage/
├── __init__.py
├── subPackageA/
│   ├── __init__.py
│   └── foo.py       # Has a function foo()
└── subPackageB/
    ├── __init__.py
    └── bar.py       # Has a function bar()

First case: all the __init__.py files of the package and subpackages are empty (that's important, we'll get to that later).

In this case, if we want to import the functions foo and bar from their modules foo and bar, we would have to do the following:

>>> from myPackage.subPackageA.foo import foo
>>> from myPackage.subPackageB.bar import bar

Nothing new here.

Now second case, we update myPackage/subPackageA/__init__.py and put this little piece of code:

import os

current_dir = os.path.dirname(os.path.realpath(__file__))
__path__.append(os.path.join(
    os.path.abspath(os.path.join(current_dir, '..')),
    'subPackageB'
))

With that in place we can do some magic:

>>> from myPackage.subPackageA.foo import foo
>>> from myPackage.subPackageA.bar import bar

Note that we imported bar from subPackageA, but bar.py is still under myPackage/subPackageB:

>>> from myPackage.subPackageA import bar
>>> bar.__file__
'<...>/myPackage/subPackageB/bar.py'

But what happened here? Why were we able to import bar under subPackageA's namespace just by adding a piece of code inside myPackage/subPackageA/__init__.py? Time for some explanations.

The `init.py` file and the usage of the `path` attribute

The __init__.py file is a special file that indicate that the current folder is a Python package. During the import process, when Python finds the source of a module or package it is asked to import, it will read and execute all the __init__.py files of the package and subpackages of the module.

So take the following structure. Each of the __init__.py file has a simple print(__name__) so that we can see what's going on.

packageA/
├── __init__.py
└── packageB/
    ├── __init__.py
    └── packageC/
        ├── __init__.py
        └── packageD/
            ├── __init__.py
            └── module.py

And when we try to import the module:

>>> from packageA.packageB.packageC.packageD import module
packageA
packageA.packageB
packageA.packageB.packageC
packageA.packageB.packageC.packageD

So in our example above, the code we added in subPackageA/__init__.py we executed when Python executed from myPackage.subPackageA.foo import foo. This piece of code look perhaps a bit cryptic, but it simply does one thing: add the absolute path to myPackage/subPackageB/ to the __path__ of subPackageA.

The __path__ attribute of a Python package is used by the import system when trying to import modules of subpackages. It is a list (or at least an iterable) of paths that can be used when importing modules or subpackages.

For example:

>>> import django
>>> django.__path__
['<...>/venv/lib/python3.9/site-packages/django']
>>> import django.db
>>> django.db.__file__
'<...>/venv/lib/python3.9/site-packages/django/db/__init__.py'

So when we updated subPackageA.__path__ to add subPackageB, we told Python to bind the objects of subPackageB/ under the namespace of subPackageA, which is why we were able to import bar from subPackageA.

Two points to note though:

The update of subPackageA.__path__ can be done even if the __path__ has not been updated with the new path yet. I believe this is because when Python tries to resolve the paths of the imports, it executes all the __init__.py files it finds in paths that matches (even partially) the import required. I still needs confirme this though.
It is still possible to import the objects of subPackageB under its own namespace.

Possible usage of this trick

Now that you know this is possible, you may be wondering what could be a possible real-life use of this.

__path__ can be used sometimes when you're dealing with dynamic imports. For example Django uses them when dealing with modules provided by the user.

Another possibility is to hide the internal structure of a package. If you can dynamically choose which namespace to bind your objects to, you can basically choose the structure you want for your package, no matter the internal organizations of files and folders. Note that this usage is probably just theoretical, as it wouldn't be really useful in real life. I see a few reason for this limitation:

Like I said before, the dynamic binding of objects to a namespace other than their own doesn't prevent them to be imported via their own namespace (cf. point 2. above).
When a package is installed into the site-packages folder (whether system-wide or in a virtualenv), it is possible to see the source files of it, so people would still be able to figure out how the package is structured internally.
If you really want to acheive this, it might be easier to add imports to the __init__.py files, as they are executed when the package is imported, than to mess with __path__ attribute.

A last possibility is to be able to extend packages via some extensions that can be installed separately, and registered to the main package, to make their usage as transparent as possible. Let's take an example to make things a bit easier to understand.

Let's say you create a package imageLib to handle image manipulation. You have functions to resize, crop, rotate... All this functions needs to have an image decoded (ie. a matrix of pixels) to work, so you need to provide functions for encoding and decoding each common image format (png, jpg, gif...). But you don't want to include all the extensions into your lib, so that people will have to download and install only those they need.

That's where __path__ manipulation comes handy. The extensions can have a .register() function that will find the __path__ attribute of the lib (or any of it's subpackage), and add themselves to it, so that the import can be as smooth as possible.

Turns out this ability is something that is quite popular, named namespace packages. This was a functionnality available using the pkgutil package, but it's part of Python since Python 3.3 (Sept. 2012) with PEP 420. This PEP is really interesting, but that'll be for another article.

DEV Community: Bastien ANTOINE

Debugging Python programs without an IDE

What is pdb?

How to use it?

Using run() inside a REPL

Setting a breakpoint with set_trace()

Using the builtin breakpoint() function

Interacting with pdb

Note on the syntax

l(ist)/ll

c(ont(inue))

q(uit)

p/pp

n(ext)/s(tep)

w(here)

u(p)/d(own)

r(eturn)

a(rgs)

interact

ipdb, the evolution

Note on running ipdb with breakpoint()

Some drawbacks

An interactive terminal is required

The interface is not very user-friendly

Sources

Discovering Python Namespace Packages

What is a package in Python?

The specificity of the namespace packages

Differences with the regular packages

Step by step basic example

Dynamic loading of package parts

Limitations

Possible usage

More concrete example

Sources

Http Requests Cheatsheet

Selecting the HTTP method

requests (Python)

cURL

Headers

requests (Python)

cURL

Authentication

HTTP Basic

requests (Python)

cURL

HTTP Bearer

requests (Python)

cURL

Query parameters

requests (Python)

cURL

Body parameters

requests (Python)

cURL

Sending files

requests (Python)

cURL

Disabling SSL verification

requests (Python)

cURL

3XX redirects

requests (Python)

cURL

Logging

requests (Python)

cURL

Quick intro to Bash I/O redirections, *here documents* and *here strings*

stdin, stdout and stderr

Controlling the program input and output

Controlling the input

Here documents and here strings

Here strings

Controlling the output

Controlling stdout

Controlling stderr

A note on the pipe operator

Sources

Playing with Python `__path__` attribute

The __init__.py file and the usage of the __path__ attribute

Using `run()` inside a REPL

Setting a breakpoint with `set_trace()`

Using the builtin `breakpoint()` function

`w(here)`

`u(p)/d(own)`

`ipdb`, the evolution

Note on running ipdb with `breakpoint()`

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

`requests` (Python)

Quick intro to Bash I/O redirections, here documents and here strings

`stdin`, `stdout` and `stderr`

Controlling `stdout`

Controlling `stderr`

Playing with Python `path` attribute

The `init.py` file and the usage of the `path` attribute