Forem

Joash Xu
Joash Xu

Posted on • Originally published at enzircle.com on

1 1 1 1 1

Go producer-consumer problem

Carlos Aranda via unsplash

In the producer-consumer problem, a producer repeatedly generates data and writes it into a buffer. The consumer repeatedly reads the data in the buffer, removing it from the buffer and using the data in some way.

In this particular problem, the producer reads in tweets from a stream, and a consumer is processing the data to find out whether someone has tweeted about Golang or not. The task is to implement a producer and consumer that run concurrently to increase the throughput.

Solution

Key takeaway

The use buffered channel

You can opt to implement this using an unbuffered channel. When no goroutine reads the channel, the execution will pause. So this will decrease throughput. The length of the buffer depends on your need. For this problem, I just use ten. The buffered channel is declared at line 34.

Clean up your goroutines

When you launch a goroutine, make sure that it will eventually exit. Go runtime can not detect that a goroutine will never be used again. If a goroutine does not exit, the scheduler will still periodically give it time to do nothing, which slows down your program. This is called a goroutine leak.

In our solution, we clean up our goroutine on line 12. Once the stream is closed, we also close the link channel.

Range and close

You can read from a channel using a for-loop construct. It will keep reading from the channel until the channel is closed.

So in our solution, line 20 will keep reading the channel link. Until that channel is closed (at line 12).

Wait for the process to finish

In our solution, line 43, the execution will pause until a value is written to the done channel. And we write to this channel at the end of our consumer function (line 27) to indicate that we are done. If you don't wait, the program will exit immediately.

Wrap up

So in this article, we again see Go concurrency's building block, namely goroutines and channels. The following article will examine a more traditional means of writing concurrent code through memory access synchronization and the primitives that follow that technique.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (1)

Collapse
 
b0r profile image
b0r

Hi Joash,
can you please also provide the source code in a text format so I can copy/paste it, instead of retyping everything from the image?

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay