DEV Community: Antonio Morrone

Iterators & Generators in Python

Antonio Morrone — Sun, 27 Dec 2020 15:54:22 +0000

This post aims to describe the basic mechanisms behind iterators and generators.

Iterator protocol

As in many programming languages, Python allows to iterate over a collection. The iteration mechanism is often useful when we need to scan a sequence, operation that is very common in programming.
In Python the iterator protocol involves two components: an iterable and an iterator.

Iterable

The iterable is the container through which we want to iterate. It is the object that needs to be scanned to retrieve all the elements (or some of them). Some of well-known iterables are lists, tuples, dictionaries, ranges. In the iterator protocol the iterable exposes a iter method that returns an iterator object.

Iterator

The iterator is the data structure that allows the scan through the container. It could seem a complication, but actually, with the separation of concerns, it lets the developer separate the concept of container, from the concept of iteration. The container object doesn't need to keep the state of an iteration and furthermore, on the same object, many iterations at the same time can take place, so keeping the iteration state in a different object is a must. The container is a collection of elements, while the iterator is kind of an handler of the container, it exposes the same elements (owned by the container) one-by-one with a specific order and . In the iterator protocol, the iterator exposes two methods: iter and next. While the first one return the object itself (it allows the usage of both container and iterator in for and in statements), the latter return the next item from the container. What does it make the iterator end the iteration? The StopIteration exception.

Iterable and Iterator Examples

Below an example of iterator protocol implementation:

And its usage:

Generator

Generators are methods with yield statements. The yield statement has the power to suspend the function execution and store its state, so that it can be resumed. Behind the scenes, python returns the control to the function caller and save the function state; in this way, at the next execution, the function will start where it left, without letting the developer worrying about the state of the function. Generators ARE iterators, but not vice-versa.

Here an example of generator:

and its usage:

Async generator

Like generators they are async function with a yield statement.

Conclusions

Iterators are iterables.
Iterators are objects that implement the iterator protocol consisting in implementing both __iter__ and __next__.
Iterator iterations stop when StopIteration is raised.
Generators are methods with yield statements.
Async generators are async methods with yield statements.
Whenever possible, generator must be the preferred method due to its simplicity, while the protocol implementation gives much more control.

Resources

Mongodb Query Profiler

Antonio Morrone — Sat, 10 Oct 2020 16:16:18 +0000

Some months ago, I was creating a web page to show some aggregated data, but I soon noticed the API used to retrieve the data was very slow. After investigating on the possible issue, we discovered the bottleneck: the database. The solution was to re-structure the data to make it consumable from a web page.

Although I had used sometimes the SQLServer profiler, I had no experience about MongoDB profiler, so here below the steps involved to analyse a MongoDB query.

It doesn't apply to our case, but in many situations performance problems are related to missing or wrongly implemented MongoDB Indexes.

Enable MongoDB profiler

By default, MongoDB doesn't profile any query. It is possible to set the profiling level executing:

db.setProfilingLevel(<profile_level>)

where profile_level can be:

0 - the profiler is off, so it doesn't collect any data.
1 - the profiler collects data only if the operation takes more than slowms (a configurable threshold).
2 - the profiler collects data for all operations.

IMPORTANT: At the end of the analysis, please remember to disable profiling with: db.setProfilingLevel(0), since that it can affect MongoDB performance.

Retrieve last query profile

Once profiling is enabled, it is possible to retrieve last query profiled (meaning that the query must be executed after setting the profiling level) with the command:

db.system.profile.find().limit(1).sort({ts:-1}).pretty()

Explain

The explain command comes in handy, since that it provides information on many common operations such as aggregate, count, distinct, find, findAndModify, delete and update. Nevertheless a method with the same name, is available on collection and cursor also.

So if we want to analyse the information associated with an aggregate operation we can execute:

db.<collection_name>.explain(<verbosity>).aggregate(...)

where:

<collection_name> is the name of the collection on which the aggregation is performed
<verbosity> is the level of verbosity we want to extract.

Explain verbosity

The explain command accepts one of the following verbosity levels:

queryPlanner - It returns the queryPlanner information about the winning plan evaluation.
executionStats - It returns queryPlanner and the executionStats information, but rejected plans are not included in the latter.
allPlansExecution - It returns queryPlanner and the executionStats information about all the evaluated plans (including also rejected plans).

Conclusion

Only by understanding how the query was executed by MongoDB, we were able to improve its performance.

Not our case, but I would remind that performance problems are often associated to missing or wrongly implemented indexes. If so, please have a look at MongoDB Indexes.

Resources

Browser Extension Development

Antonio Morrone — Sat, 10 Oct 2020 16:07:09 +0000

Web extensions or browser plugins can be used to improve browser functionalities. The following examples summarize the different goals they can have:

no-script - to block script served from not-trusted domains
ublock Origin - it is "a wide-spectrum blocker"
dark-mode - it changes the appearance of web pages
React Developer Tools - to inspect react-specific properties of a page that makes use of React.

Anatomy of an extension

The main components of a browser extension are:

Manifest.json - it is the only mandatory file and it describes the extension itself. It is as an entry-point for the browser to declare what are the files/resources/permissions will be used.
Background scripts - They contain the logic that are not strictly related to the single web-page. They are loaded when the extension is loaded and remain active until the extension is disabled or uninstalled.
Content scripts - Unlike background scripts, they are loaded into web pages, so they can manipulates DOM exactly like normal script can do.
UI components - They includes all the parts that permit a user to interact with the extension, by means of an HTML document. There are three different UI components:
Web-accessible resources - They include all the resources that are made available to the scripts (e.g. HTML, images, jss, css)
Extension pages - They are additional pages used to handle specific user interactions.

Web-extension toolbox

As for many browser APIs, also the web extension APIs can behave differently according with the browser on which they are executed. In case of cross-browser development it could be handful web-extension polyfill developed by Mozilla.

Furthermore, to ease the development process, there is also web-extension toolbox and its related yeoman generator. Among the others it provides the following functionalities:

Compiles the extension via webpack to dist/<vendor>
Watches all extension files and re-compiles on demand
Reloads extension or extension page as soon something changed

After answering to some questions, it generates a ready-to-run web extension.

Development

You only need to run npm run dev <vendor_name>, where can be chrome, firefox, opera or edge; now you can load the generated extension in the browser of choice.

Build

As easy as running npm run build <vendor_name>.

Conclusions

So, what are you waiting for? Are you ready to develop your own browser plugin? I am planning mine ;-)