DEV Community

Dead Simple Python: Generators and Coroutines

Jason C. McDonald on August 01, 2019

Like the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press. Programming is often about waiting....

Read full post

Florimond Manca • Aug 2 '19 • Edited

Very in-depth article about generators! I enjoyed it a lot.

At first your use of the term "coroutine" when referring to generators that use .send() and yield from was a bit jarring to me — as of Python 3.6 a coroutine is the return value of a coroutine function:

async def foo():
    pass

print(type(foo())  # coroutine

But then I realized that you were probably using that term as the more general computer science concept of a routine that can be paused during execution (see Coroutine).

Still, the fact that coroutine is now "reserved terminology" in Python might be confusing to some people. Perhaps a disclaimer that coroutine refers more to the computer science general concept rather than the coroutine built-in type would be helpful. :-)

Jason C. McDonald • Aug 2 '19 • Edited

Well, no, not precisely. In Python, the term "coroutine" does indeed officially refer to both. In fact, the two have their own qualified names.

What I described is called a simple coroutine, which was defined in PEP 342, and further expanded in PEP 380. Coroutines first appeared in Python 2.5, and continue to be a distinct and fully supported language feature.

You're referring to a native coroutine (also called an asynchronous coroutine), which was defined in PEP 492, and was based on simple coroutines, but designed to overcome some specific limitations of the former. Native coroutines first appeared in Python 3.5. Again, this didn't replace simple coroutines, but rather offered another form of them specifically for use in concurrency.

I'll put a little clause or two about this in the article.

Also, don't worry, I'll be coming back around to async and concurrency soon; once that's written, I'll come back to this article and link across.

Florimond Manca • Aug 2 '19

Thanks for clarifying :) Actually, I wasn’t aware that native coroutine was the official name for generators used in this fashion.

I'll put a little clause or two about this in the article.

Thanks! Just to be clear, I was simply raising the concern that as async programming is becoming more and more used/popular in Python and most people talk about coroutines as a shorthand for async coroutines, using the shorthand to refer to native ones could be confusing. Anyway I think you’ve got the point so thanks for taking that into account. :)

Jason C. McDonald • Aug 2 '19

Uh oh! I just realized I'd had a dyslexic moment, and read something in PEP 492 backwards...

What I described are simple coroutines, and the newer type is the native coroutine (also called an "asyncronous coroutine").

Blinks

Naming is hard.

Anyhow, I've gone back and edited both my comment and article. Thanks again...if you hadn't asked about that, I would have never caught my error!

rhymes • Aug 2 '19

Great article Jason!

Just a couple of details:

An exception can be raised at the current yield with foo.raise(). -> with foo.throw().

In sorted(self.letters.items(), key=lambda kv: kv[1]) the lambda can be replaced with operator.itemgetter(1), it's one of my favorite small things that are in the standard library :D

I was wondering if there was a way to simplify the coroutine code, using a context manager. The __enter__ could call send(None) and the __exit__ could call close().

With a simple generator is easy to do something similar:

>>> from contextlib import contextmanager
>>> @contextmanager
... def generator():
...     try:
...             yield list(range(10))
...     finally:
...             print("cleanup...")
...
>>> with generator() as numbers:
...     print(numbers)
...
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
cleanup...

But the same doesn't work for a coroutine...

As a first I came up with this:

from contextlib import closing

def print_char():
    try:
        while True:
            print(f"char: {yield}")
    finally:
        print("cleanup...")

with closing(print_char()) as printer:
    printer.send(None)
    for c in "hello world":
        printer.send(c)

>>>
char: h
char: e
char: l
char: l
char: o
char:
char: w
char: o
char: r
char: l
char: d
cleanup...

I came up with something like this then:

from contextlib import ContextDecorator

class coroutine(ContextDecorator):
    def __init__(self, function):
        self.coro = function()

    def __enter__(self):
        self.coro.send(None)

    def __exit__(self, exc_type, exc, exc_tb):
        self.coro.close()

    def send(self, *args):
        self.coro.send(*args)



def print_char():
    while True:
        print(f"char: {yield}")

printer = coroutine(print_char)
with printer:
    for c in "hello world":
        printer.send(c)

but I'm not sure it's improving much :D

Jason C. McDonald • Aug 2 '19 • Edited

Ooh! Thanks for catching that typo! That would have been confusing.

As to the lambda or itemgetter(), I'd actually gone back and forth between the two in writing that example. I think using the lambda there is my own personal preference more than anything.

That is certainly a clever combination of a context and a coroutine, by the way. (Naturally, I didn't discuss contexts in this article, as I haven't discussed them yet in the series.)

Thanks for the feedback.

Abdur-Rahmaan Janhangeer • Aug 3 '19

The way it presented generators, i knew it would be a 👌 read, and i was right!

Taking the time to cover only those two helps a lot, best article on coroutines i've read to date. Rhanks for writing this up 👍