DEV Community: Ruben Sanchez

69 tips I got from The Pragmatic Programmer

Ruben Sanchez — Mon, 02 Aug 2021 05:49:42 +0000

The Pragmatic Programmer is a book by Andrew Hunt and David Thomas written in 1999. I have been reading it over the last few months and my final impression is that I am sure that any programmer out there can learn something by reading it. Novice or experienced, this book is packed with good advice. I have tagged the very best tips with a chili pepper (🌶), for reference.

If you enjoy this article, get the book

So, what do you need to do to become a Pragmatic Programmer?

1) Early adopter/fast adapter

2) Inquisitive

3) Critical thinker

4) Realistic

5) Jack of all trades

6) Cares about the craft

7) Never runs on auto pilot

8) Pragmatic Programmers think beyond the immediate problem, always trying to place it in its larger context 🌶

9) They take responsibility for everything they do

10) Isn't afraid to admit ignorance or error 🌶

11) You need to have a broad base of knowledge and experience to pull all of this off. Learning is a continuous and ongoing process

12) When you accept responsibliity for an outcome, you should expect to be held accountable for it

13) Before approaching anyone to tell them why something can't be done, is late or broken, stop and listen to yourself. Will they ask "have you tried this..." or "didn't you consider that?..." How will you respond? 🌶

14) We need to decide what to test at the unit level. Typically programmers throw a few random bits of data at the code and call it tested. We can do much better using the ideas behind design by contract. This style of testing requires you to test subcomponents of a module first. Once the subcomponents have been verified, then the module itself can be tested.

15) By designing code to pass a test and fulfull its contract, you may well consider boundary conditions and other issues that wouldn't occur to you otherwise.

16) Constantly review what's happening around you, not just what you personally are doing 🌶

17) Many users would rather use software with some rough edges today than wait a year for the multimedia version

18) You must invest in your knowledge portfolio regularly

19) Having the best ideas, the finest code or the most pragmatic thinking is ultimately sterile unless you can communicate with other people 🌶

20) When you are faced with an important meeting, jot down the ideas you want to communicate and plan a couple of strategies for getting them across

21) You are communicating only if you are conveying information. To do that, you need to understand the needs, interests and capabilities of your audience 🌶

22) Make what you are saying relevant in time, as well as in content. Sometimes all it takes is the simple question "Is this a good time to talk about...?"

23) Programmers are constantly in maintenance mode

24) Imposed duplication: Developers feel they have no choice - the environment seems to require duplication.

25) Inadvertent duplication: Developers don't realize that they are duplicating information

26) Impatient duplication: Developers get lazy and duplicate because it seems easier. 🌶

27) Interdeveloper duplication: Multiple people on a team duplicate a piece of information

28) Bad code requires lots of comments

29) We all know that in the heat of the moment with deadlines looming, we tend to defer the updating of documentation

30) Appoint a team member as the project librarian, whose job is to facilitate the exchange of knowledge

31) Two or more things are orthogonal if changes in one do not affect any of the others

32) We want to design components that are self contained: independent and with a single well defined purpose. When components are isolated from one another, you know that you can change one without having to worry about the rest.

33) You get two major benefits if you write orthogonal systems: Increased productivity and reduced risk 🌶

34) Single components can be designed, coded, unit tested and then forgotten. There is no need to keep changing existing code as you add new code.

35) An orthogonal approach promotes reuse and will probably be better tested, because it will be easier to design and run tests on its components.

36) When teams are organized with lots of overlap, members are confused about responsibilities. Every change needs a meeting of the entire team, because any one of them might be affected.

37) If I dramatically change the requirements behind a particular functions, how many modules are affected? In an orthogonal system the answer should be "one"

38) Avoid global data 🌶

39) Keep your code decoupled 🌶

40) Avoid similar functions

41) Building unit tests is itself an interesting test of orthogonality. Do you have to drag in a large percentage of the rest of the system just to get a test to compile? If so, you've found a module that is not well decoupled from the rest of the system.

42) The first part of any time estimation is building an understanding of what is being asked. Estimates given at the coffee machine will come back to haunt you. 🌶

43) Be strict in what you will accept before you begin, and promise as little as possible in return.

44) When debugging, don't gloss over a routine or piece of code involved because you "know" it works. Don't assume it - Prove it. 🌶

45) Is the problem being reported a direct result of the underlying bug, or merely a symptom? Is this bug really in the compiler? Or is it in the OS? Or is it in your code? If you explained this problem in detail to a coworker, what would you say? If the suspect code passes its unit tests, are the tests complete enough? What happens if you run the unit test with this data? Do the ocnditions that caused this bug exist anywhere else in the system?

46) You could convince yourself that the error can't happen and ignore it. Instead, Pragmatic Programmers tell themselves that if there is an error, something very bad has happened.

47) A dead program normally does a lot less damage than a crippled one. 🌶

48) Will this code still run if I remove all the exception handlers? If the answer is "no", maybe exceptions are being used in non-exceptional circumstances.

49) The routine or object that allocates a resource should be responsible for deallocating.

50) Deallocate resources in the opposite order to that in which you allocate them.

51) A good way to stay flexible is to write less code. 🌶

52) Do you depend on the "tick" coming before the "tock"? Not if you want to stay flexible.

53) Always design for concurrency

54) Separate views from models

55) Fred doesn't know why the code is failing because he didnt know why it worked in the first place. Always be aware of what you are doing. Fred let things get slowly out of hand, until he ended up boiled like the frog. 🌶

56) Attempting to build an application you don't fully understand or to use a technology you aren't familiar with, is an invitation to be misled by coincidences. 🌶

57) If you don't have fundamentals or infrastructure correct, brilliant bells and whistles will be irrelevant.

58) Don't be a slave to history. Don't let existing code dictate future code 🌶

59) All code can be replaced if it is no longer appropriate

60) Next time something seems to work, but you don't know why, make sure it isn't just a coincidence.

61) Rather than construction, software is more like gardening 🌶

62) Don't try to refactor and add functionality at the same time.

63) Make sure you have good tests before you begin refactoring. Run the tests as often as possible. That way you will know quickly if your changes have broken anything.

64) Test your software or your users will

65) No matter which "best practices" it includes, no method can replace thinking. 🌶

66) Good requirements documents remain abstract

67) The team speaks with one voice -- externally -- Internally we encourage lively and robust debate.

68) Ask yourself if the software meets the performance requirements under real world conditions -- with expected number of users or connections or transactions per second -- Is it scalable?

69) On documentation, we like to see a simple module level header comment, comments for significant data and type declarations and a brief per class and per method header

Must-Bookmarks to become good in Go

Ruben Sanchez — Thu, 01 Apr 2021 09:14:10 +0000

I am having some days off, so I have compiled a list of Golang resources to go through during this time, aimed to become a better Go developer. I am hoping that this can also be helpful to you. By the time we all reach the end of the list, our level of language familiarity should have increased a lot!

Fundamentals

How to write Go code

Quick initial refresher in case you need it!
https://golang.org/doc/code

Effective Go

I think it's a very good idea to revisit the "Effective Go" document once in a while with different skill level. In my case, the understanding I get from it and the profitability is nothing compared to the understanding I got a while ago when I read it for the first time. The more your mind has been exposed to different Go codebases, the more insight you can find when reading this.
https://golang.org/doc/effective_go.html

Code Review comments

I discovered this document recently and as the description says, it's a nice complement for the Effective Go write up. It's full of interesting comments that you should take into account, useful when doing code reviews (or writing your own code!)
https://github.com/golang/go/wiki/CodeReviewComments

Standard library

Not being an extremely large standard library, getting to know it quite well is quite advisable. Having a pretty good knowledge will save you from the error of reaching for a third party library to accomplish something that can be easily done remaining within the stdlib.
https://golang.org/pkg/#stdlib

Also, remember that the language specification is here to help
https://golang.org/ref/spec

Tricky parts

Pointers

Getting more into action, I got to admit pointers were hard to grasp at first. If you want an explanation which is comprehensive and very easy to follow, check Junmin Lee's fantastic video
https://www.youtube.com/watch?v=sTFJtxJXkaY

Benchmarking and profiling

I am almost sure that if you are doing Go, one of the main reasons is because you want peformant code. So well, things don't always go as expected, and when issues arise, knowing how to benchmark and profile will give you a big advantage. This comprehensive workshop contains a lot of interesting information about this matter.
https://dave.cheney.net/high-performance-go-workshop/gophercon-2019.html

Tips, tricks and common gotchas

Even though Go has a pretty short language specification that doesn't mean it's an easy language. I think Go is in fact difficult... or at least difficult to get right. It's really easy to get "whatever" up and running and I think that explains its popularity, but if your goal is not just to have "whatever" but idiomatic code which is precise, clean and simple to understand then you better be ready to face a pretty nice collection of quirks that you should know in order to write the best possible code. Here are two good articles on this topic.

The darker corners of go
https://rytisbiel.com/2021/03/06/darker-corners-of-go/
50 shades of go
http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in-go-golang/

Project based learning

Sometimes you want to get a good understanding of many crucial parts (mocks, interfaces, etc...) without doing toy exercises or easy tutorials. In this case the most enjoyable option is to do a long project tutorial like the following:

Http servers with ScyllaDB and Kafka https://www.youtube.com/watch?v=IS9Erqkx-ks&list=PLGRs9mrW62D1I4SF1p03zNn0x2WSSFhOA

You can also find a collection of guided long-ish tutorials here:

Gophexercises https://gophercises.com/

Golang University

Almost ending the list, here is a triad of playlists put together by John Arundel from Bitfield Consulting which can serve as a good general resource, available in three different skill levels.

BASIC
MEDIUM
ADVANCED

Books

Ending the list, it's time for books. I like books because the signal to noise ratio is so much better than many online tutorials. Unfortunately in some cases they might get outdated very easily, but if you pick books with the right content, many times the learnings you get from them should be timeless. For instance, the original "The Go Programming Language" book might not have the latest and greatest information about modules, but that is something that you can research on your own and doesn't make the book less valid as the best starting point for the language.

Having said that, a couple of titles that have grabbed my attention

The interpreter/compiler book

The Interpreter Book and its sequel, The Compiler Book are two resources to accomplish writing an interpreter (and a compiler) for a made up language. Really interesting proposition and definitely on my "to do" list.
https://interpreterbook.com/
https://compilerbook.com/

Practical Go Lessons

This excellent resource almost replaces entirely the rest of the list. A mamooth-sized book with everything you need to know about writing Go. The HTML version of the book is free, but you can support the author buying paper and digital copies. A must-bookmark!
https://www.practical-go-lessons.com

Go RabbitMQ to Kafka ETL with zero lines of Go!

Ruben Sanchez — Sat, 16 Jan 2021 19:00:12 +0000

I think that for a big percentage of cases, ETL pipelines are a solved problem. Unless you have very special needs, there are many tools in the market that can remove the hassle of setting up irrelevant minutia, and let you focus on the business logic instead, because anyone can create a function that connects to a Rabbit MQ exchange, but only you can work on the task of how your business logic should handle the received messages. However, I am aware of the value that "reinventing the wheel" might have in some business aspects, when the off the shelf tools won't cut it. But in the event that off the shelf tools are good for you, let me introduce you to Benthos.

What is Benthos

Benthos it's a tool that can perform streaming ETL tasks, so to speak, it can grab your stream source (Extract), apply some transformations (Transform) and send it to your desired sink/destination (Load). It's written in Go, and sorry for the clickbait title, sure it has a lot of Golang lines of code! But the whole point of this exercise is that it doesn't require you to write any in order to set your pipelines. It uses a yaml config file which is executed by the main binary.

Benthos handles for you some boring stuff such as parallelism, or creation of metrics in Prometheus, Influx, Cloudwatch, etc.. We are going to see how it behaves with the following task:

Getting a message from a Rabbit queue
Manipulating the payload
Sending the new payload to a Kafka topic

Getting started

For this exercise I have needed a docker-compose file so I could test Rabbit and Kafka in my local environment. If you were to deploy a Benthos job in an environment with an already existing instance of Rabbit and Kafka, you wouldn't worry about this, but let me share my docker-compose.yml file just in case

version: "3.6"
services:

  rabbitmq:
    image: 'rabbitmq:3.7-management-alpine'
    hostname: rabbitmq
    expose:
      - 5672
      - 15672
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: "guest"
      RABBITMQ_DEFAULT_PASS: "guest"


  zoo1:
    image: zookeeper:3.4.9
    hostname: zoo1
    ports:
      - "2181:2181"
    environment:
        ZOO_MY_ID: 1
        ZOO_PORT: 2181
        ZOO_SERVERS: server.1=zoo1:2888:3888


  kafka1:
    image: confluentinc/cp-kafka:5.5.1
    hostname: kafka1
    ports:
      - "9091:9091"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zoo1

  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka1:19091"
    depends_on:
      - kafka1

  benthos:
    image: jeffail/benthos
    expose:
      - 4195
    ports:
      - "4195:4195"
    volumes:
      - ./pipeline:/pipeline

Do a docker-compose up -d and let it start as long as needed, Rabbit is notoriously slow starting off. In the meantime let's create our Benthos pipeline.

The pipeline

You might have seen how in the previous docker compose file we mounted ./pipeline:/pipeline in the Benthos image. This is just because that's where I am going to store my pipeline file. Feel free to mount a different directory if you need.

So here are the contents of pipeline/benthos.yml

input:
  amqp_0_9:
    url: amqp://guest:guest@rabbitmq:5672
    queue: "ruben"

pipeline:
  processors:
    - sleep:
        duration: 500ms
    - bloblang: |
        root.original_event = this
        root.new_attr = "abc"
        root.new_attr2 = 123

output:
  kafka:
    addresses:
      - kafka1:19091
    topic: ruben
    client_id: benthos_kafka_output
    key: ""
    partitioner: fnv1a_hash
    compression: none
    static_headers: {}
    max_in_flight: 1
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""

The input and output sections are self explanatory, they just define the configuration of the brokers we are using. The middle section (pipeline) is a bit more involved, because it requires the usage of the custom Benthos components to manipulate messages. See all the available processors here

In my case, I am using the bloblang processor. You can see the details here. I am getting the input messages and applying a manipulation to their payload.

So that's it, that's the pipeline. I understand this is a toy example, but if you look at the documentation you can see how Benthos seems to be ready to tackle pretty serious cases.

Running the example

Let's start with Rabbit. I didn't really add a custom json configuration to this docker compose stack, so there are no custom queues when initializing. We can simply go to http://0.0.0.0:15672/#/queues and create one (in my example, it's called ruben, as you can see in the benthos.yml file)

Now we can do the same in Kafka. Open Kafdrop at http://0.0.0.0:9000/topic/create and create a topic (I also named it ruben)

Now we can now run Benthos with our config file:

docker-compose run benthos -c ./pipeline/benthos.yml

If everything goes well, Benthos will start listening. Now,
send a message (or as many as you want) via the Rabbit interface. Remember that your processor in Benthos is now ready to handle json objects.

Go to the Kafdrop interface and observe your messages arriving. Indeed, you have the original message, plus the two extra attributes that we have defined in our pipeline.

Conclusion

While perhaps it doesn't match everyone's use case, Benthos looks like a pretty nice and easy to use alternative to create ETL pipelines in the Golang universe.

The entirety of the code is in this article, but if you need it to be in a GitHub repo, here it is https://github.com/rubenwap/poc-benthos

Create a useful Alexa skill with minimal Python knowledge

Ruben Sanchez — Tue, 08 Dec 2020 15:18:11 +0000

I am pretty disappointed with the terrible quality of most of the "custom" stuff found in the Alexa skills shop. Lots of "tell me a joke", "tell me a fun fact", "tell me random facts", "play forest sounds", "play sea sounds"... So at the end, my Echo unit ended up being used just to turn Netflix on and off and manipulate the heating temperature.

That's no more! It's very easy to create custom skills with Python (or Node.js) which can actually be useful so you find yourself triggering them often. Because after all, how many times can you play "random facts"?

How to find out what is actually useful

In my case, I focused on trying to remember small actions that I might have repeated day after day using either my phone or the computer. Mechanical tasks that might not be pleasurable or be considered as leisure, but you need to do them anyhow. Consider if you find yourself often checking the same bit of information.

Bike hire status

I live in Barcelona and here we have a public bike hire system called Bicing. Nowadays with working from home I don't use it that much, but when I need it, I used to open the app in my phone before leaving home in order to find out whether there are available bikes in the station near my home. Bicing has a live status page in JSON format, so that was the perfect candidate for my Alexa Skill.

Getting started

First you must create an free account if you don't already have one, and log in to the Skills dashboard

Click on "Create Skill" when you are ready and follow the steps to create a custom skill from scratch with Python (you can select Node.js, although the example on this tutorial will be Python)

You should end up with an empty skill in your dashboard

Configuring the skill

There are two critical items you should understand:

INVOCATION

Invocation is what name or phrase should you tell Alexa so she understands that you want to communicate with your skill.

For example, I have a skill to control my heating temperature, so in the sentence "Alexa, tell the heating to lower the temperature one degree", the heating (or heating, since Alexa seems pretty flexible with grammar particles) is the Invocation. Or "Alexa, tell Xbox to play Grand Theft Auto". Xbox would be the Invocation.

So in my case I am using Estación de bicing, which translates to "bicing station" (I am creating the skill in Spanish).

INTENT

So, how many actions do you intend your skill to handle? example intents would be (for my heating) "set temperature", "switch on and off", "change mode"... For my bike hire status skill I am just going to do one intent "AreThereBikes"?

Add as many intents as you want in the dashboard

And for every intent, you need to add the sentences that would trigger it. In my case I have put a list of sentences such as "Are there bikes", "are there bicycles", "are there available bikes"... and son on. Think of the possibilities.

Adding your code

Once your code is configured with Invocation and Intent, do save and build the language model. You can find those buttons in the Intent screen. You need to rebuild every time you make changes to it.

Click on the "Code" tab in the upper part of the page. You can edit your code here. You can also do it in VSCode or your favourite local editor, but the set up is a bit longer, so we will stick here.

I have added this class to the lambda_function.py file:

class HayBicisIntentHandler(AbstractRequestHandler):
    """Handler for HayBicis Intent."""
    def can_handle(self, handler_input):
        # type: (HandlerInput) -> bool
        return ask_utils.is_intent_name("HayBicis")(handler_input)

    def handle(self, handler_input):
        # type: (HandlerInput) -> Response
        bikes_available = get_bikes(244)
        speak_output = bikes_available

        return (
            handler_input.response_builder
                .speak(speak_output)
                .response
        )

The structure of the class will be done for you automatically. You just need to change a couple of things. In my case, I have put the proper intent name:

return ask_utils.is_intent_name("HayBicis")

and have defined that my handle method should respond with the speak_output created by:

bikes_available = get_bikes(244)
speak_output = bikes_available

So what is this get_bikes(244) function? This is the very simple custom code I have included in the skill. It makes a request to the Bicing JSON live status and returns a string with the information I want Alexa to repeat.

def get_bikes(station_id):
    resp = requests.get("https://api.bsmsa.eu/ext/api/bsm/gbfs/v2/en/station_status")
    available_bikes = list(filter(lambda item: item["station_id"] == station_id, resp.json()["data"]["stations"]))[0]["num_bikes_available_types"]
    return f"""Hay {available_bikes["mechanical"]} bicis mecánicas y {available_bikes["ebike"]} eléctricas."""

The code retrieves the json and then, filters the response so it only uses the stations where the station_id matches with the id we have provided. Out of that selected station, it grabs the amount of available bikes.

Last point, don't forget to register your intent at the bottom of the file, as such:

sb = SkillBuilder()

sb.add_request_handler(LaunchRequestHandler())
sb.add_request_handler(HayBicisIntentHandler())
sb.add_request_handler(HelpIntentHandler())
sb.add_request_handler(CancelOrStopIntentHandler())
sb.add_request_handler(SessionEndedRequestHandler())

Testing

So that's the full story. Of course we still need to add some other niceties, such as persisting user settings so anyone could select their own station (right now it's hardcoded to the station where I live!) But we can go step by step in future tutorials.

Now is time to test. After you have saved your code, make sure you click on the "Deploy" button in the upper right side of the code editor, and then click on the "Test" tab.

Here you can choose whether using your microphone or simply type the questions. With this testing you will see if Alexa reacts as you would expect:

Great! That is exactly the outcome we wanted. We now have a working Alexa skill which we have made in barely a few minutes.

Deploying to the Alexa store

I won't deploy this project to the store yet, as it's missing a crucial part for it to be useful for someone else (setting your own station). But if you were inclined to deploy your skills, you can click on the "Distribution" tab and get started with the process. Good luck!

GitHub repo

https://github.com/rubenwap/haybicis

Making a fast "yes" clone with Go

Ruben Sanchez — Sun, 18 Oct 2020 12:59:33 +0000

Following the series on creating Go based clones of the GNU coreutils, today I wanted to do a speed experiment using yes

What is YES?

If you are using mac or linux, go to your terminal and type yes. You should get an infinite stream of y. Or you can also run it as yes yes and you will get an infinite stream of yes. Or yes no.

Graphical representation of this tool:

What is the point of this?

Imagine that you have a long installation script that you want to run unattended. If the script has a flag to force positive answers for every question, then no issue. But what if the script really forces you to wait for the questions and answer y or n every time? What if you don't care because you know you will answer y to every question? Then you can do:

$ yes | myInstallScript.sh

So a constant stream of y will be piped into your install script, so whenever you are asked about something, the pipe will answer for you.

Another use is if you need to produce dummy data, you can pipe yes to a file for as long as you need.

So... while true, print yes?

Kind of, but not entirely, because we will focus on performance. If we didn't care, we could do:

expletive := os.Args[1]
if expletive == "" {
    expletive = "y"
}
for {
    fmt.Println(expletive)
}

And that would be the end of the story.

However, let's measure how fast this implementation is. Please go ahead and install pv if you are using mac, via brew install pv. On linux you probably already have it. On Windows you probably need to use Cygwin. Pv is a tool to measure the data through your pipe.

Implementation 1: Infinite loop print

go build yes.go
./yes | pv > /dev/null

This prints an average of [1.31MiB/s]

Considering that the real yes installed on a mac (the BSD implementation) does around [23.4MiB/s], the conclusion is that our first implementation is not very good.

Implementation 2: Why print just one when you can print 100 million

So far we have been doing

for {
    fmt.Println(expletive)
}

which doesn't seem very efficient because it's just printing the loop one by one. Let's spice it up by printing huge blocks of the repeated string:

fmt.Println(strings.Repeat(expletive+"\n", 100))

That gives us [89.8MiB/s]. Much better! We have already passed the benchmark of the original yes installed on macs.

Why stop at 100?

fmt.Println(strings.Repeat(expletive+"\n", 100000000))

That gives [1.37GiB/s] We seem to be onto something here. Beyond 100 million repeats, at least on my computer two things can happen: Either there is no difference, or it simply doesn't work and writes don't begin.

Implementation 3: Goroutines assemble!

One would think that goroutines can help here. If yes is being thrown at high speed by multiple goroutines, won't that give high throughput? I don't know! We will discover!

The problem is that if we do simply like this:

yesChan = make(chan string)
go func() {
    yesChan <- strings.Repeat(expletive+"\n", 100000000)
}()
go func() {
    fmt.Println(<-yesChan)
}()

We will run out into an interesting problem (after a few seconds running):

panic: too many concurrent operations on a single file or socket

Which is a self-explanatory issue. The solution for that is to use a semaphore. However, I am worried that the code is getting way too complicated for the simple goal that we want to achieve:

maxWorkers = runtime.GOMAXPROCS(0)
sem = semaphore.NewWeighted(int64(maxWorkers))

ctx := context.TODO()

for {
    if err := sem.Acquire(ctx, 4); err != nil {
    log.Printf("Failed to acquire semaphore: %v", err)
    break
    }

    go func() {
        defer sem.Release(2)
        yesChan <- bytes.Repeat([]byte(expletive), 100000000)
    }()

    go func() {
    defer sem.Release(2)
    os.Stdout.Write(<-yesChan)
    }()
}

So, this must be the bomb, right? Let's see how super fast it goes....

(...suspense...)

[1.42GiB/s]

WHY? Why is it practically the same as in the previous serial operation with 100 million writes?

Well, the answer is simple: The improvement is barely noticeable, because even if coroutines are parallel operations, I/O operations are not. You cannot have two writes to the same output at the very same time. This must be a serial operation, that's why the overcomplicated goroutines code is not helping here.

Implementation 4: Bytes are our last hope

Notice that so far we have been using strings. Let's change it to bytes

yesChan = make(chan []byte)
go func() {
    defer sem.Release(2)
    yesChan <- bytes.Repeat([]byte(expletive), 100000000)
}()
go func() {
    defer sem.Release(2)
    os.Stdout.Write(<-yesChan)
}()

Do we have any improvement? Yes! (but not a lot) [1.63GiB/s] So, working with bytes instead of strings really helped.

Implementation 5: Back to the basics

We have agreed before that we don't need the goroutines/semaphore mess, so we can bring this script back to basics:

for {
    os.Stdout.Write(bytes.Repeat([]byte(expletive), 100000000))
}

That's all we need to achieve 1.63GiB/s!

Also, consider that your mileage might vary, because the kind of computer you use to run this also plays a role. I have tried to run other people's code reported to give about 4GiB/s and I have been getting barely the half of it, so think about that if you want to run this on your machine.

Implementation 6: Will a bytes buffer help?

Short answer: At least with my medium knowledge of it, it doesn't seem to.

for {
    b.Write(bytes.Repeat([]byte(expletive), 100000000))
    b.WriteTo(os.Stdout)
}

That gives [1.28GiB/s]

Implementation 7: What about a proper bufio buffer?

Yes! This is the fastest I can reach with the processor and memory that I use:

for {
    f := bufio.NewWriterSize(os.Stdout, 8192)
    f.Write(bytes.Repeat([]byte(expletive+"\n"), 100000000))
}

This gives [1.73GiB/s] I have been unable to achieve more so far, if you have a hint to make this better, let me know.

Full version of implementation 7 (the fastest)

This is the full code. Most of it is boilerplate related to the command line flags, the meaty part is very small.

package main

import (
    "bytes"
    "github.com/urfave/cli"
    "os"
    "bufio"
)

func yes() *cli.App {
    app := cli.NewApp()
    app.Name = "yes"
    app.Usage = "yes outputs expletive, or, by default, ``y'', forever"
    app.Action = func(c *cli.Context) error {
    expletive := c.Args().Get(0)
        if expletive == "" {
            expletive = "y"
        }

        for {
            f := bufio.NewWriterSize(os.Stdout, 8192)
            f.Write(bytes.Repeat([]byte(expletive+"\n"), 100000000))
        }
    }
    return app
}

func main() {
    app := yes()
    app.Run(os.Args)
}

Will this quick Julia lesson make you forget about Python for your data needs?

Ruben Sanchez — Fri, 02 Oct 2020 17:52:24 +0000

I have been a Python heavy user for some time, so when I learned about Julia I thought that the prospect it was introducing was very interesting. Simplicity in the syntax but execution power closer to C, but maybe the obstacle to beat was too high. Python already has massive adoption, there is a huge ecosystem with modules for any task you can think about and even people from the R world seem to be migrating to it. Also, Matlab seems to keep having strong presence in Academia. So, what is the incentive to change? Will Julia arise and become a main actor in the scientific computing community?

Indeed, it is already quite relevant if you look at the case studies you will see how companies such as NASA, CISCO or PFIZER are already using it in serious projects.

While I don't suggest to throw all your Python project out the window and rewrite everything in Julia, I think the language is interesting enough to merit a quick look at its functionality. So if you have heard comments on unstability or packages not mature enough, you can draw your own conclusions after giving it a try.

Installation

After you install Julia from their site (or if you are on mac, doing brew cask install julia), I am going to assume that you also want a notebook environment. For that you have two options. You can install the desktop app Nteract, or if you are so inclined, stick with the classic Jupyter Notebook (or Jupyter lab) for which you need to have Python installed, so you can do pip3 install jupyterlab.

When you have your notebook application, you still need the actual Julia notebook kernel, so go to your terminal and type julia. You should see this:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.1 (2020-08-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>

Now type the ] symbol, and the prompt will change to this:

(@v1.5) pkg>

So you can type

(@v1.5) pkg> add IJulia

press enter, and observe how the kernel package gets installed. If you had already opened your notebook before, it's pretty likely that you need to close it and open it again so it detects the new Julia package.

Getting started with the language

The basic things are what you might expect:


# Printing
println("Hello")

# Variables
my_name = "Ruben"

typeof(my_name) # String

# Arithmetic
sum = 3 + 7
difference = 10 - 3
product = 5 * 5
quotient = 100 / 10
power = 10 ^ 2
modulus = 101 % 2

# String interpolation
interp = "My name is $my_name"

Some interesting quirks:

# Note that single quote denote a character
typeof('a') # char
typeof("a") # string

# Concatenations can force transformations
# Like in this case, from int to string

string(10, " can also be converted")
# outputs: "10 can also be converted"

Some nice structures:

# Dictionary
phonebook = Dict("Ruben" => 555234898, "Santa Claus" => 00358787876)

# Tuple
animals = ("dog", "donkey")

animals[1] # Outputs "dog" because Julia is 1 indexed!!!!

# Mix types
farm2 = ["chicken", "hen", "horse", "cow", 56]

Control structures and functions

# classic while. What you would expect
n = 0
while n < 10
    n += 1
    println(n)
end

# the for is quite pythonic
for n in 1:10
    println(n)
end

Now something new. Look at this way to create vectors, either populated or empty. We can do it either with random data or specific data

# a 5 rows 3 cols vector with random data
r, c = 5,3
A = rand(r, c)

# same but with zeroes
Z = zeros(r, c)

Because of this:

# We can have a for loop with two conditions in that vector
for i in 1:m, j in 1:n
    A[i,j] = i + j
end

# Or something looking like a Python list comprehension
C = [i + j for i in 1:m, j in 1:n]

Finally, functions can be defined in different ways

function hi(name)
    println("Hi $name")
end

hi2(name) = println("Hi $name")

hi3 = name -> println("Hi $name")

# and you would call them as you expect
hi("world")
hi2("world")
hi3("world")

What about dataframes

If you are going to work with dataframes, you are going to need to know how to install (and import package). We have already seen one way to install (when we added IJulia earlier on)

Another way is from within your code:

using Pkg
Pkg.add("CSV")
Pkg.add("DataFrames")

Although you might want to do that with the other option (via terminal) to don't execute this every time you run your code. Normally when you ship a Julia project you would have a .toml file with all the dependencies, so you can grab them at once by issuing the ] instantiate instruction.

Once you have installed the DataFrames module (and CSV if you need it), we can create Dataframes from files or from our own data constructions. For the example, I am using this file

df = DataFrame(CSV.File("sp500.csv"))

You can use pretty familiar functions to get the heads and lasts:

# 3 is how many rows you want to display
head(df, 3)
first(df, 3)
last(df, 3)

You can get the column names

names(df)

And select specific columns

df["column_name"]

Selecting rows uses the array index for what you need. For example, this is the first 5 lines and the first 3 columns

df[1:5, 1:3]

Or you could get very specific searching. Just show me the rows where the symbol is either MMM or ABT

filter(row -> row.symbol in ["MMM", "ABT"], df)

To summarize

Although by this article you could think that the biggest plus of Julia is that it has a very nice syntax, the truth is that I am only scratching the surface, since its real benefit is way deeper than that. It's not only a matter of speed (but also)

Take a look at the article on Why We Created Julia to have a clearer view.

Julia seems a good solution for the two languages problem with Python (if you want speed you need to revert to C extensions), it has modules for most tasks you might need, specially in the scientific computing area (including ML) and it feels stable and well maintained.

If you are a happy Python/R/Matlab user, you might not switch overnight, but I still think it's a very fresh and interesting proposition.

Upskill yourself by recreating GNU Coreutils in Go

Ruben Sanchez — Sat, 26 Sep 2020 21:10:52 +0000

Reinventing the wheel for fun

One of the most popular advice to gain skills programming is to reimplement tools that already exist. For instance, if you wanted to learn how to do a CRUD API, I am pretty sure you probably practiced with a "To do" app. The point of this is not to create something new and groundbreaking, but to use well known ideas to improve what you know. Other examples you might have seen are things such as implementing a chat server to learn about websockets, a weather app, a Hacker News reader...

Today I am talking about something I consider more interesting: Reimplementing the GNU Coreutils utilities. I am not the first person doing this, but I am convinced that you are going to come out of this exercise with way better skills in your language of choice.

So what are the Coreutils? If you use Linux or Mac, you have probably used a bunch of them. Perhaps on Windows too in some cases. Take a look at this link for more information.

You can approach this project in different ways. In my case, I know that the point of coreutils is that they are really small and really simple, they do just one task but they do it well. In the original repo in C, most of the tools are just one single file. I am going to start with the same idea.

Understand the functionality reading the man page. Requirements.
Implementing that functionality freely with no guidelines other than my own criteria. I do the project plan!
Comparing the output of my tool against the output of the real coreutil tool. This is the QA part.

As you see, this is more like a very free exercise rather than a serious structured practice. I am also very flexible in terms of whether or not to use extra libraries. For example, I realized that the flag package in Go is a little bit tedious to set up. Other flag packages are much nicer. Also, I need to find one that let concatenate arguments such as -wl instead of -w -l, to get as close as possible to the real Coreutils functionality.

WC

The first tool I picked was the word counter WC. Not the most difficult, not the easiest. For me, the requirements looked like this:

Individual functions for every kind of count
Flag parsing
Main function that would get the input from Stdin and files passed as argument
Depending on the passed flags, the counting functions would run, using the text from the previous step as argument
Results are printed on screen

So my rough schema:

And here is the code. As mentioned before, the only external dependency is for my own personal taste, but could be easily avoided (specially when I recently spoke about unnecessary dependencies here!) But if we take this as an initial exercise, I think the result is not bad.

I have commented the code so you can follow along

package main

import (
    "bufio"
    "bytes"
    "fmt"
    "github.com/urfave/cli"
    "io/ioutil"
    "log"
    "os"
    "strings"
    "unicode/utf8"
)

/*
We define the four basic counting functions. 
The interesting thing with Go is that unlike 
languages like Python, len(string) will give you 
the byte count, not the character count. 
For characters, you need to count runes. 
*/

func byteCounts(text string) int {
    return len(text)
}

func lineCounts(text string) int {
    return len(strings.Split(text, "\n"))
}

func characterCounts(text string) int {
    return utf8.RuneCountInString(text)
}

func wordCounts(text string) int {
    return len(strings.Fields(text))
}

/* 
We initialize the CLI App with the definition 
of the flags. I think that the way this library 
handles flags is pretty clean and elegant. 
But feel free to use the `flag` package too. 
*/

func wc() *cli.App {
    app := cli.NewApp()
    app.Name = "WC"
    app.Usage = "The wc utility displays the number of lines, words, and bytes contained in each input file"
    app.Flags = []cli.Flag{
        cli.BoolFlag{
            Name:  "c",
            Usage: "The number of bytes in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "l",
            Usage: "The number of lines in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "m",
            Usage: "The number of characters in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "w",
            Usage: "The number of words in each input file is written to the standard output",
        },
    }

// We define what actually happens when running

    app.Action = func(c *cli.Context) error {

        var buf bytes.Buffer
        var m map[string]int
        m = make(map[string]int)

/* 
We check for text passed to Stdin, just so we 
can invoke the app by doing `echo "hello" | wc -m` 
*/

        scanner := bufio.NewScanner(os.Stdin)
        for scanner.Scan() {
            buf.WriteString(scanner.Text())
        }

        if err := scanner.Err(); err != nil {
            log.Println(err)
        }

/* 
Maybe some filenames have been passed as argument. 
We have to retrieve the text from those and 
store it in a buffer 
*/

        for i := range c.Args() {
            fmt.Print(i)
            content, err := ioutil.ReadFile(c.Args().Get(i))
            if err != nil {
                log.Fatal(err)
            }
            buf.WriteString(string(content))
        }

// Now, depending on passed flags, we run the counting

        if c.Bool("c") {
            m["clen"] = byteCounts(buf.String())
        }

        if c.Bool("l") {
            m["llen"] = lineCounts(buf.String())
        }

        if c.Bool("m") {
            m["mlen"] = characterCounts(buf.String())
        }

        if c.Bool("w") {
            m["wlen"] = wordCounts(buf.String())
        }

        if c.Bool("c") && c.Bool("m") {
            m["mlen"] = 0
        }

// Printing the final values

        for _, value := range m {
            if value != 0 {
                fmt.Print("\t", value)
            }
        }
        fmt.Println("")

        return nil
    }
    return app
}

func main() {
    app := wc()
    app.Run(os.Args)
}

Is not perfect, and the functionality is not 100% on par with the real tool, but I learned a couple of things while doing this. That was the goal!

Just for fun, here is the goroutines/channels enabled version. Pick your choice!

package main

import (
    "bufio"
    "bytes"
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "strings"
    "sync"
    "unicode/utf8"

    "github.com/urfave/cli"
)

var wg sync.WaitGroup

func byteCounts(text string) int {
    return len(text)
}

func lineCounts(text string) int {
    return len(strings.Split(text, "\n"))
}

func characterCounts(text string) int {
    return utf8.RuneCountInString(text)
}

func wordCounts(text string) int {
    return len(strings.Fields(text))
}

func wc() *cli.App {
    app := cli.NewApp()
    app.Name = "WC"
    app.Usage = "The wc utility displays the number of lines, words, and bytes contained in each input file"
    app.Flags = []cli.Flag{
        cli.BoolFlag{
            Name:  "c",
            Usage: "The number of bytes in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "l",
            Usage: "The number of lines in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "m",
            Usage: "The number of characters in each input file is written to the standard output",
        },
        cli.BoolFlag{
            Name:  "w",
            Usage: "The number of words in each input file is written to the standard output",
        },
    }
    app.Action = func(c *cli.Context) error {

        var buf bytes.Buffer
        var m map[string]int
        m = make(map[string]int)

        scanner := bufio.NewScanner(os.Stdin)
        for scanner.Scan() {
            buf.WriteString(scanner.Text())
        }

        if err := scanner.Err(); err != nil {
            log.Println(err)
        }

        for i := range c.Args() {
            fmt.Print(i)
            content, err := ioutil.ReadFile(c.Args().Get(i))
            if err != nil {
                log.Fatal(err)
            }
            buf.WriteString(string(content))
        }

        bytesChan := make(chan int)
        linesChan := make(chan int)
        charactersChan := make(chan int)
        wordChan := make(chan int)

        wg.Add(4)
        go func(text string) {
            bytesChan <- byteCounts(text)
            wg.Done()
        }(buf.String())
        go func(text string) {
            linesChan <- lineCounts(text)
            wg.Done()
        }(buf.String())
        go func(text string) {
            charactersChan <- characterCounts(text)
            wg.Done()
        }(buf.String())
        go func(text string) {
            wordChan <- wordCounts(text)
            wg.Done()
        }(buf.String())

        for i := 0; i < 4; i++ {
            select {
            case msg1 := <-bytesChan:
                m["c"] = msg1
                m["m"] = 0
            case msg2 := <-linesChan:
                m["l"] = msg2
            case msg3 := <-charactersChan:
                m["m"] = msg3
                m["c"] = 0
            case msg4 := <-wordChan:
                m["w"] = msg4
            }
        }

        wg.Wait()
        for key, value := range m {
            if c.Bool(key) && value != 0 {
                fmt.Print("\t", key, ": ", value)
            }
        }
        fmt.Println("")

        return nil
    }

    return app
}

func main() {
    app := wc()
    app.Run(os.Args)
}

Running the final version

We can build the executable:

go build -o gwc wc/wc.go

Where gwc is my desired executable filename (to don't mix it up with the real wc) and wc/wc.go is the path where I keep the script. On mac and linux you will get the gwc file you need to call with ./gwc unless you add it to your path. On Windows you should get gwc.exe.

And we can run it:

echo "hello dev.to" | ./gwc -c

outputs 12

If you are on Windows, I am not sure how passing stdin text works in the command line, so you could try this alternate example:

Create a file with some text, and do:

gwc.exe -c myfile.txt

So now, time to implement the other 97 remaining coreutils!

I was doing bad stuff in my first dev job. Sorry! (Part II)

Ruben Sanchez — Thu, 24 Sep 2020 12:22:49 +0000

Glad to see you back. It means that you liked the previous part of this article. Or if you haven't seen it yet, please check it out.

I would like to continue the series by explaining about some more dubious techniques I was using during my first dev job. I hope you can find some advice and good tips from reading it!

Bad dependencies

So in that first job, we were very keen on researching new cool tools. I specially remember a case where we had to use a famous library for Python, but for some reason I stumbled upon a another library that was internally using the previous library to accomplish the very same thing, but with a couple of nice improvements on top.

The outcome of the story? I lost quite a long time because shortly after implementing it in my project, things started to fail and initially it wasn't obvious that it was because of the library (if you remember the first part of the article, I wasn't fond of logging important stuff back then). When I finally realized that this was the cause, it also took quite a bit of time to find out about the specific reason that caused the problem. After spending all that time, we realized the wrapper was not even maintained very frequently, so a pull request that someone had done with that fix just stayed in the PR limbo lingering forever.

Choosing dependencies is not a trivial task. Is not enough with finding something that does what you want. You also have to consider whether it will be maintained or perhaps it will be abandoned and issues forgotten forever. It's true that open source software has tools to fix that (you can usually fork and fix the stuff yourself if nobody is caring about the main repo), but you have to analyze your cost/benefit if you are doing this.

Also, in some cases you have to consider about whether you really need a dependency. Have you read about the Left pad debacle? Please go ahead and read it, it's very interesting.

Structure and discipline

So, consider this, you work in a team of 3 people and you all work in the same language (let's say Python). Jim likes to bootstrap his projects with Pipenv. Mike uses Poetry and Scott just uses the Python global environment. Mike adds a setup.py, Jim likes to keep all the files in the root folder and Scott puts as much code as possible inside a single file, so it's easier to find later. Mike builds his Docker based projects with the official Python full image. Jim uses Python slim and Scott uses the Ubuntu image.

Now, the manager has notified the guys that Brandon will join the team next Monday and would like the team to give him an overview on how to build Python projects. What's the next step here?

The existing members of the team have to agree on a unified way to work. And the way they need to do it is ensuring that Brandon can navigate the projects easily and intuitively from day one. You need a project structure in which placements of the parts are obvious. You shouldn't need people to tell you where things are.

If you cannot agree, you can use well established public layouts such as:

GitHub is not a storage room

By the time I discovered about version control, I ran into the usual issues that beginners encounter when starting to use it.

Not using branches: I realized quickly that juggling different attempts for features plus fixes for the current version inside the master branch was going to develop into a tragic mess of revert commands. Please use branches.
Not learning beyond the basics: So your muscle memory can already do git status git add -A git commit -m "Changes" and git push origin master. Nice, but that is the happy path. I advise to learn about how to react when problems arise (merge conflicts) so you are used to work with things like git mergetool and understanding how can you revert to previous commits.

However, the most striking error when using Git and GitHub, is the fact that some beginner developers would treat it as the final storage area where you stick your finished projects. So then your code is not in GitHub because "is not ready". Well, that's the whole point of GitHub. When you bake a cake you put it in the fridge when you are finished, but your code is not a cake and GitHub is not a fridge. In fact we could say that GitHub is the kitchen itself. Please, commit early and commit often.

Don't learn this

Surprisingly I found out that knowing how to do something is not really a very interesting skill. Let's say you work with Python. Well, knowing how to open a CSV file is not interesting. Anyone can Google how to do that. Knowing how to connect to a database is not interesting. Same as before, all those are just skills taken from the instructions manual of certain libraries. Can you create a dataframe out of json data? Very few people will care. Specific skills (how to use this or that library to accomplish this or that task) are not very valuable. If you don't know them today and you need them, just read the documentation.

However, the really powerful thing is to take all what you have read in the docs and put it together in a way that accomplishes what I describe in this series of articles. That will indeed leave a good footprint and make you a well rounded developer. To be able to "use Python" is not as amazing as the fact of being able to use it well and efficiently. It will really make a difference.

I was doing bad stuff in my first dev job. Sorry! (Part I)

Ruben Sanchez — Thu, 24 Sep 2020 11:30:17 +0000

I got my first developer job almost by accident. I was not looking for a dev job back then, but while I was working doing something else, some tools I had developed as work side project got some traction and I was able to get an offer from the same company to move into a position to develop full time.

That was pretty cool! But looking back, I did things that would send shivers down the spines of security professionals, devops guys and other kinds of reasonable people. Luckily it served to teach me a valuable lesson, so let me share some of what I did, in case you find it useful.

FIRST PART

Testing (or lack thereof)
Deploy like it's 1997
Monitoring what matters

Testing (or lack thereof)

When I started programming I was into Chrome Extensions. I loved the simplicity of sticking bits of DOM manipulation into a package and see them coming to life upon page load. The first extension I did professionally was interacting with an internal tool of the company and it got pretty popular, so the team using it started to request a lot of very diverse functionality.

For me, every new feature was as simple as building the correct javascript using the browser's developer console, sticking the finished bit somewhere inside the extension script and call it a day. You can imagine that after many feature requests the extension grew to be a messy monster. Calling it spaghetti code doesn't do it justice. Copy-Paste-Oriented-Development could be an alternative too.

After I left the team, the new maintainers struggled with this project. What earlier was managed fully inside my brain -not the best scenario-, was now a meaningless bunch of files with lots of undocumented functions with names that kinda made sense only for some people. Because of that, new feature requests on that extension are now on hold.

The fact that you know 100% that it works is not enough, because even in the most solitary project you can do, some day you will leave the company and the new people won't have a clue about why it should or shouldn't work. Besides that, can you guarantee that your brain will process all the possible scenarios for the feature to go through?

A test won't slow you down. What will slow you down is to waste one day of work to find out why your new feature is breaking the previous feature. Testing is not some super special technique you must learn separately from coding. Testing IS part of coding.

Helpful links:

For Javascript: I don't do a lot of JS recently, so my recommendation might be outdated, but last time I tried, I was having an easy time with Mocha
For Python: Even though Python has unittest bundled in, I like to use Pytest
For Golang: For unit testing, you are more than covered with the standard Go testing library. For cases when you want BDD testing, you can look into Ginkgo

(If I haven't convinced you and you are still not testing your code, at the very least write detailed documentation!)

Deploy like it's 1997

Apparently nowadays is very cool to hit a button and engage a complex pipeline that will test your code, package it and deploy it. Even your infrastructure lives in a piece of code so it can be replicated.

Back when I started working with this it wasn't 1997, but in my mind it was, because my deployments consisted on connecting to a remote server and copy paste some files from my local computer. I know that FTP'ing stuff was the norm a while ago, but on top of that I used to edit the remote copy to add bits of code (which I had not saved in my local...), had credentials files laying around everywhere, and during a long time (luckily fixed after a while) the whole Python ecosystem on that machine ran off a Global Python environment without virtual envs, so services needing different dependency versions were out of luck. There was a bit of everything, a couple of web apps, some services, etc...

If you code is good enough to be shipped, ship it and don't touch it. Docker images are ideal to preserve an untouched environment. If you want to tinker with something because you forgot to fix a typo, release it again. Ideally your release should be automated so it's repeatable, but I am going to say that at the very least, release it from your local, don't do remote edits. Editing your production copy of something is a really bad idea, not because your typo fixing is dangerous in itself, but because it sets a precedent that can escalate to worse cases.

Monitoring what matters

So when you start having lots of processes up and running, some are bound to fail. What will you do when something goes wrong? So ideally you would have a nice setup to monitor what matters, like I describe in this article. Depending on where are you deploying your tools, you could pair it with a nice logging tool. I like Google Operations (formerly known as Stackdriver). Every cloud has its own and there are some other cloud agnostic tools out there.

So what did I do when my processes started to fail for the first time? Well, I had a genius technique which consisted on re-running the failed step and stare at the console to observe the crash. Naturally nothing came up because I was not logging anything. I couldn't help but starting with print(suspiciousVariable). It would be some time until I upgraded to use a logging library.

You need to do that. You will be in control of a more flexible logging format and will be able to pick multiple outputs if needed. Please, do not print stuff, log it instead. And most importantly, log the relevant tracebacks and items that will help you diagnose a problem. Don't be me that time when after a failure I just logged the text "it failed". Very helpful. We could also talk about using a debugger, but that's for a future article.

If you found this interesting, come back for the second part, where I will talk about:

Bad dependencies
GitHub is not a storage room
Project structure

And will also advise on something you definitely don't have to learn.

Monitor the behavior of your Python app by learning InfluxDB, Grafana and Telegraf

Ruben Sanchez — Sun, 13 Sep 2020 10:53:11 +0000

We will go through a tutorial where Python and Flask will be used. However, you will soon realize that Python and Flask are the less important parts here. Due to the nature of the setup (a docker-compose file) you shouldn't have any trouble swapping the Docker for my Flask app with whatever else you want to monitor.

So, what are we actually doing here? What is the final goal?

We will create docker compose file which will instantiate the following services:

A web app of your own that you want to monitor
An instance of Telegraf, which is a tool that can send all sorts of metrics about your app's performance to a database.
An instance of InfluxDB, which is the database where all the Telegraf info will go to, and also where you will write your own metrics from your application.
Grafana, which will serve to create an awesome dashboard and display all that information.

Look at the power of Grafana by visiting their demo here

The duo Telegraf/InfluxDB might get confusing. You could consider that Telegraf will record into InfluxDB data about your performance (cpu usage, memory usage, etc...) But if you want to record things related to the content of your app (requests made to a certain endpoint, how many times a function has been called, how many times it throws an error...) you need to actively save that into InfluxDB from your application. We will see how.

Getting Started

You may clone my repo if you want to follow this easier, or build it yourself as we go:

https://github.com/rubenwap/metrics-stack

Setting up the project

Your app

Go to the folder that contains the app you want to use for this (or look at the cloned repo). The important thing is that you should have a working Dockerfile that if you run it, it launches your app standalone.

Now go ahead and create an empty docker-compose.yaml file.

version: "3"
services:
    metricsweb:
        container_name: metricsweb
        build: .
        ports:
        - 5000:5000
        depends_on:
        - influxdb

As you see, we start by adding one service: Our app. In my case, it's a Flask app, so I am opening the port 5000. This service depends on the future service influxdb. Notice the build parameter. For it to work, you need to have a Dockerfile in the same directory. Mine looks like this:

FROM python:3.7.4-slim-stretch
COPY . /app
WORKDIR /app
EXPOSE 5000
RUN pip install -r requirements.txt
ENTRYPOINT python /app/main.py

Grafana

Grafana is an open source analytics and visualizations solution which is compatible with a lot of databases. In this example we are using InfluxDB since it's really quick and optimized to store time series data, but for example you make Grafana read from your SQL database.

For more information, visit their website

This is the relevant bit you need to add in the docker-compose:

    grafana:
        container_name: grafana
        image: grafana/grafana:latest
        ports:
        - 3000:3000
        volumes:
            - ./grafana/data:/var/lib/grafana

Pretty straightforward. Grafana uses the port 3000 for the web app. We will configure this later.

InfluxDB

Take a look at the Influx website so you will get an idea about its features. In order to add it to our stack, we need this additional piece inside our docker-compose file:

    influxdb:
        container_name: influxdb
        image: influxdb:latest
        ports:
        - 8086:8086
        env_file:
            - 'env.influxdb'
        volumes:
            - ./influxdb/data:/var/lib/influxdb

Some important things:

I am mounting a local directory volume to store the data. This is so data persist even if you destroy the stack and recreate it again (I did the same with Grafana in the previous step)
I am reading some variables from an env file. You could also insert them straight into the yaml file, it makes no difference. The most important variable is INFLUX_DB, which tells you the name of the DB that will be created when the stack goes up. Here is the file I am using:

INFLUXDB_DATA_ENGINE=tsm1
INFLUXDB_REPORTING_DISABLED=false
INFLUX_DB=metrics
INFLUXDB_USER=admin
INFLUXDB_ADMIN_ENABLED=true

Telegraf

Telegraf is made by the same company that makes InfluxDB. You can read more about that here It's a server agent with a lot of plugins that can connect to InfluxDB and store many metrics about your application: CPU usage, memory usage, etc... You can definitely extend what it can detect.

This is the relevant docker-compose bit:

telegraf:
        container_name: telegraf
        image: telegraf
        container_name: telegraf
        restart: always
        depends_on:
        - influxdb
        environment:
            HOST_PROC: /rootfs/proc
            HOST_SYS: /rootfs/sys
            HOST_ETC: /rootfs/etc
        volumes:
            - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - /sys:/rootfs/sys:ro
            - /proc:/rootfs/proc:ro
            - /etc:/rootfs/etc:ro

Notice that besides the volumes that we mount to have persisting data, we have some environment variables telling Telegraf where to get certain information about processes and system.

For the volumes part to work, you will need to create a telegraf.conf file in your local directory with some settings. Here you can see the contents. It's pretty basic and surely it can be improved.

To summarize, here is the full docker-compose.yaml that you will need:

https://github.com/rubenwap/metrics-stack/blob/master/docker-compose.yml

Add code for your metrics

You are ALMOST done. If we would bring up the stack right now, everything would work fine, except the fact that your application wouldn't be sending anything to Influx and you would only have access to system data. Let's add something:

Consider that I only have a main.py file which is a very simple Flask web app, this is what I have:

from flask import Flask
from datetime import datetime
from influxdb import InfluxDBClient


client = InfluxDBClient('influxdb', 8086, 'admin', 'admin', 'metrics')
app = Flask(__name__)

@app.route("/hello")
def hello():
    client.write_points([{
        "measurement": "endpoint_request",
        "tags": {
            "endpoint": "/hello",
        },
        "time": datetime.now(),
        "fields": {
            "value": 1
        }
    }])
    return "Hello World!"


@app.route("/bye")
def bye():
    client.write_points([{
        "measurement": "endpoint_request",
        "tags": {
            "endpoint": "/bye",
        },
        "time": datetime.now(),
        "fields": {
            "value": 1
        }
    }])
    return "Bye World!"

if __name__ == "__main__":
    app.run (host="0.0.0.0")

See how we first initialize a client for the database with the proper parameters, and then on each endpoint, we call the method to write into the database with the relevant content. I have chosen to record a "1" every time the endpoint is called, so later I can visualize that in the dashboard by requesting the sum of that field. Not the only way to do this, but just one way.

Up and running

Build with docker-compose

Run this in your console to bring the stack up:

docker-compose up --build

(to bring it up later on, omit the build argument if you don't have anything new to rebuild)

Your stack would now run on interactive mode (you will see the logging appearing in your console). You can stop it by pressing Ctrl+C. However, by doing so the stack doesn't fully go down. You would then need to do

docker-compose down

If you want to launch it detached from the console, do

docker-compose up -d

Configuring the dashboard

Ok, so your stack is up and running. Make some calls to your api endpoints so it registers some visits and there is some data recorded in the database. Then open Grafana by going to http://localhost:3000

How to use Grafana

After login (admin/admin), first thing you need to do is to add a data source:

You need to configure InfluxDB using influxdb as a host name, because that is the name of the service in the docker compose file. Otherwise we would need to inspect the container to find the internal Docker IP address and that's more convoluted because it can change. Using influxdb to autodiscover the current IP will save us the trouble.

Create a new dashboard...

And you are ready to add new panels. For this example I am going to add the percentage of idle CPU. I have this data courtesy of Telegraf. Since this is a trivial app, you will see that the percentage is huge, because I am doing nearly nothing with this app:

If you click on the lower part of the screen, in the query builder, you can select your FROM (in this example we only have one database) and that should autopopulate the dropdown menus with the available selections.

Look at this other panel. This creates a counter of the amount of times /hello has been visited. This data comes from the influxdb code that we have put into our Python app

Look how endpoint_request appears in the dropdown, since that is the key we are saving from our code. Then you can choose what to do with this data. In my case I present the sum of all entries in a counter.

As you see, Grafana is a really cool and comprehensive tool. I recommend you to play around building panels for different metrics. It's really powerful!

I hope you could find some use from this tutorial. Grafana and Influx are also a new subject for me, so if you have any tips or comments, let me know!

How do you approach your DEV "Reading List"?

Ruben Sanchez — Fri, 22 May 2020 08:49:19 +0000

I have noticed that when faced with articles that are either long, or likely to be useful in the future, people tend to add them to their reading list.

I am curious about how do you use this list. Do you set a time during the week when you go through it and read what you have saved during the last days?

Do you use it as a sort of library, where you can go and search for answers on interesting topics whenever the need comes?

Personally, I am guilty of saving cool stuff in my reading list but then never going back to it, so I am looking for tips on how to optimize this behavior and take advantage of the awesome content I save.

Use Python in your browser client code together with JS. No server required, thanks to Web Assembly

Ruben Sanchez — Thu, 21 May 2020 10:57:53 +0000

Use case: Let't say that I love Python Pandas for data manipulation, but at the same time I need to use the manipulation results in a regular client-side webpage. The classic method I would do is to have a Python backend, do my calculations in there and then send the results to the front, via an API, server rendered templates or whatever.

I am going to present you an alternative based on Web Assembly and the Pyodide project.

Web Assembly is a standard (supported in all major browsers) that roughly speaking says "This is how executable binaries should look like and here is how they should interact with the host" (the host in this case being the browser). That opens the door to client side applications written in a wide variety of languages. Take a look at this repo with the list of available languages:

https://github.com/appcypher/awesome-wasm-langs

One of the fruits of this project has been Pyodide, which allows you to execute real Python code in the browser. Do not mistake this with other projects that allow you to write Python and then transpile it into Javascript so it can run in the browser. Here what is running is the actual Python code.

We can do a quick demo and then I walk you through what just happened:

<!DOCTYPE html>
<html>
<head>
    <script type="text/javascript">
        window.languagePluginUrl = 'https://pyodide-cdn2.iodide.io/v0.15.0/full/';
    </script>
    <script src="https://pyodide-cdn2.iodide.io/v0.15.0/full/pyodide.js"></script>
</head>
<body>
    Pass JS data to Python pandas (fully in-browser) and returns results back to JS
    <script type="text/javascript">
        const data = JSON.stringify([{ "name": "Ruben", "surname": "Sanchez" }, 
        { "name": "Michael", "surname": "Johnson" }, 
        { "name": "John", "surname": "Smith" }])
        languagePluginLoader.then(function () {
            pyodide.loadPackage(['pandas']).then(() => {
                pyodide.runPython(`
                import pandas as pd
                df = pd.DataFrame(${data})
                count = df["name"].count()
               `)
                console.log(pyodide.pyimport('count'))
            });
        });
    </script>
</body>

Save that snippet in a new html file or run it from this CodePen https://codepen.io/rubenwap/pen/yLYZORW Note that the results you are looking for will be in the console tab! last console.log in the code should print 3, and that is the value we have taken from Python.

Overall, We have done three nice things:

1) We have defined JS data to be used inside our Python script

Your starting point for this mix'n'match session can be regular Javascript data that you have obtained in any way you like:

 const data = JSON.stringify([{ "name": "Ruben", "surname": "Sanchez" }, 
        { "name": "Michael", "surname": "Johnson" }, 
        { "name": "John", "surname": "Smith" }])

2) We have executed actual Python code in the browser, including Pandas!

Don't be fooled by the ugly way in which you need to specify your Python code (as string inside the runPython method). You could write an actual nice .py file and then import the text contents via Ajax, so it would look nicer.

Note that here, loadPackage is acting somehow like an installation of that package. If you try to import pandas without having loaded it first, it won't work.


 pyodide.loadPackage(['pandas']).then(() => {
                pyodide.runPython(`
                import pandas as pd
                df = pd.DataFrame(${data})
                count = df["name"].count()
               `)

3) We have taken the results of our Python scripts and sent them back to Javascript

    console.log(pyodide.pyimport('count'))

I think this is pretty cool, the fact that you can run something in a totally different language and have it available in Javascript without having to go back and forth with API requests.

If you found this interesting, don't forget to explore their docs:
https://pyodide.readthedocs.io/en/latest/#

And generally, WASM docs, specially if you code in any of the other compatible languages:
https://webassembly.org/getting-started/developers-guide/

DEV Community: Ruben Sanchez

69 tips I got from The Pragmatic Programmer

So, what do you need to do to become a Pragmatic Programmer?

Must-Bookmarks to become good in Go

Fundamentals

Tricky parts

Tips, tricks and common gotchas

Project based learning

Golang University

Books

Go RabbitMQ to Kafka ETL with zero lines of Go!

What is Benthos

Getting started

The pipeline

Running the example

Conclusion

Create a *useful* Alexa skill with minimal Python knowledge

How to find out what is actually useful

Bike hire status

Getting started

Configuring the skill

Adding your code

Testing

Deploying to the Alexa store

GitHub repo

Making a fast "yes" clone with Go

What is YES?

What is the point of this?

So... while true, print yes?

Implementation 1: Infinite loop print

Implementation 2: Why print just one when you can print 100 million

Implementation 3: Goroutines assemble!

Implementation 4: Bytes are our last hope

Implementation 5: Back to the basics

Implementation 6: Will a bytes buffer help?

Implementation 7: What about a proper bufio buffer?

Full version of implementation 7 (the fastest)

Further reading

Will this quick Julia lesson make you forget about Python for your data needs?

Installation

Getting started with the language

Control structures and functions

What about dataframes

To summarize

Upskill yourself by recreating GNU Coreutils in Go

Reinventing the wheel for fun

WC

Running the final version

I was doing bad stuff in my first dev job. Sorry! (Part II)

Bad dependencies

Structure and discipline

GitHub is not a storage room

Don't learn this

I was doing bad stuff in my first dev job. Sorry! (Part I)

FIRST PART

Testing (or lack thereof)

Helpful links:

Deploy like it's 1997

Monitoring what matters

Monitor the behavior of your Python app by learning InfluxDB, Grafana and Telegraf

So, what are we actually doing here? What is the final goal?

Getting Started

Setting up the project

Your app

Grafana

InfluxDB

Telegraf

Add code for your metrics

Up and running

Build with docker-compose

Configuring the dashboard

How to use Grafana

How do you approach your DEV "Reading List"?

Use Python in your browser client code together with JS. No server required, thanks to Web Assembly

1) We have defined JS data to be used inside our Python script

2) We have executed actual Python code in the browser, including Pandas!

3) We have taken the results of our Python scripts and sent them back to Javascript

Create a useful Alexa skill with minimal Python knowledge