Forward hooks in PyTorch

Jan — Wed, 20 Jan 2021 12:42:51 +0000

Forward hooks are custom functions that get executed right after the forward pass. Among other things, one can use them together with TensorBoard to visualize activations of any layer.

Motivation

When using torch.nn.Module, did you ever wonder what the difference between the forward and the __call__ methods is?
One can roughly say that __call__ = forward + execution of various hooks.

Hooks?

Hooks are custom functions that get executed at specific moments during the forward/backward phase. They allow us to inspect what is going on inside of the network. The specific use cases are:

Debugging
Logging
Visualizing (this post)

If you are interested to learn more about hooks checkout the official docs.

Tell me more about forward hooks!

Forward hook is a function that accepts 3 arguments

module_instance : Instance of the layer your are attaching the hook to
input : tuple of tensors (or other) that we pass as the input to the forward method
output : tensor (or other) that is the output of the the forward method

Once you define it, you need to "register" the hook with your desired layer via the register_forward_hook method.
Once registered, the hook will be executed right after the forward method. You do not have to worry about triggering it manually!

Too vague,… I need to see an example!

I created a hands-on video tutorial where I explain step by step how to use forward hooks together with TensorBoard. The goal is to visualize activations of any layer of choice (=creating a histogram of its values for a given sample / batch).

The tutorial does not talk about several related (interesting) topics.

backward hooks
forward pre hooks

I would encourage the reader to learn more about them:)

Conclusion

Hooks are hidden gems of PyTorch. Specifically, the forward hooks allow you to debug and visualize what is going on inside of your network. This post provided a first look into what they are and how one can use them.

Credits

Cover photo https://unsplash.com/@steve_j

mltype — Typing practice for programmers

Jan — Sat, 07 Nov 2020 12:43:49 +0000

mltype is a command line tool for improving typing skills. It does so with a tiny bit of deep learning.

If you clicked on this post hoping you would learn something about static typing, type annotations or similar, this is NOT the right article. The typing I talk about in this post is the thing you do with you keyboard. Or to be precise

The action or skill of writing something by means of a typewriter or computer.

Motivation

A few months ago I decided to learn touch typing! I know what you are thinking… “Are you a faster typist than before and was all the pain worth it?” I would definitely say yes and yes. However, the internet is full of similar before and after testimonials and I am not going to write yet another one.

What I want to talk about is that I was really surprised how few resources there are for practising touch typing with programming languages. After a quick google search you will probably discover the following sites:

While the above websites have multiple strong points, let me point out some of their shortcomings

Lack of variability and element of surprise
Manual selection of source files and corresponding lines
Not customizable
Not free (typing.com)
Not nerdy enough — would it not be possible to do it in the terminal?

For the above mentioned reasons, I decided to give it a shot and write my own typing practice software: mltype.

What does it do?

In short, it is a command line tool (written in Python). It uses neural networks to generate text that looks like a programming language (or normal language). Additionally, it provides non-machine learning functionalities like reading text from a file or standard input.

If you wonder what kind of “neural network” is behind it I would more than encourage you to (re)read the The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy. mltype is doing more or less the same thing in the background. To be precise, there is a character-level language model. It spits out a probability distribution over the next character given previous characters. Most importantly, it tries to hide all the complexity and boring details of the training and inference from the user. Generating text from an existing model and training a new model can both be done in a single command.

Examples

Below are some examples of different programming languages. All the models that generated them and many other pretrained models are available for download (see the README.md on github).

Wanna try it?

If you want to know more and try it out yourself visit the below links!

github: https://github.com/jankrepl/mltype
docs: https://mltype.readthedocs.io/en/latest/

DEV Community: Jan

Forward hooks in PyTorch

Motivation

Hooks?

Tell me more about forward hooks!

Too vague,… I need to see an example!

Conclusion

Credits

mltype — Typing practice for programmers

Motivation

What does it do?

Examples

Wanna try it?