DEV Community: Odai Athamneh

The Hydra of Machine Learning: Understanding Multi-headed Attention

Odai Athamneh — Fri, 27 Oct 2023 19:18:31 +0000

🚩 TL;DR

Multi-headed attention (MTA) is revolutionizing many fields in machine learning.
The architecture uses multiple "heads", allowing the model to examine different parts of the data at the same time.
Many cutting edge AI products, like ChatGPT, are made possible by MTA.

📖 Introduction

Picture yourself trying to focus on a gripping mystery novel, but your mind keeps wandering to the pile of dishes in the sink, the email you have to send, or the news notification that just popped up on your phone. Wouldn't it be great if you could pay attention to all these different aspects simultaneously and in varying degrees?

Welcome to the world of multi-headed attention (MTA) in machine learning! This innovative approach allows machines to focus on various parts of data all at once, just like you wish your mind could.

But why is understanding this concept so crucial? Because MTA is revolutionizing fields like natural language processing, computer vision, and many more. It's the secret sauce that powers everything from chatbots to self-driving cars.

🐉 Enter the Hydra

Remember the mythical creature Hydra from Greek mythology, the multi-headed serpent that regrows two heads for every one cut off? Think of MTA as the Hydra of machine learning. Just like Hydra's multiple heads can focus on different threats simultaneously, the "heads" in MTA allow the machine to pay attention to various parts of the data at the same time.

In simpler terms, each "head" in this mechanism specializes in focusing on different aspects of the data. This enables a more comprehensive and nuanced understanding, much like how the Hydra's many heads give it an advantage in battle.

🔩 The Nuts and Bolts

First introduced in the groundbreaking paper "Attention Is All You Need" by Vaswani et al., this architecture forms the backbone of Transformer models. The above image, from that paper, contrasts it with more traditional architectures.

Here's a (very simplified) overview of how Transformers work:

Input Data: MTA starts by taking an input, like a sentence, and breaking it down into smaller parts, called tokens.
Creating Heads: These parts are then distributed across different "heads" for parallel processing.
Weighting Mechanism: Each head assigns different weights to these parts based on their importance.
Aggregation: After each head has done its job, their findings are aggregated to form a cohesive understanding of the data.
Output: The final processed data is then used for various tasks like language translation, text summarization, and more.

Even though the mechanism involves complex mathematics and algorithms, the gist is that MTA allows machines to better understand context, relationships, and nuances, making them smarter and more efficient.

🪄 Real-world Applications

The MTA mechanism is the foundation of the Transformer ML architecture. Transformers are powering some of the most cutting-edge applications in today's tech world, such as ChatGPT and DALLE.

Transformers are also seeing use in many mundane use-cases, such as automatic translation, voice assistants, and recommendation algorithms.

However, every hero has its limitations. While the Transformer architecture is undeniably powerful, it's not always the go-to choice:

Low-Resource Scenarios: Transformers are resource-hungry. In situations where computational power is scarce, like on mobile devices, these models might be overkill.
Real-Time Processing: For tasks that demand split-second decisions, the complex computations of transformers might introduce latency.

🔚 Conclusion

By now, you should have a better grasp of this marvelous technology that's quietly transforming our world. Understanding MTA isn't just for tech geeks or machine learning enthusiasts; it's for anyone curious about the technological advancements that are shaping our future.

So, the next time you talk to your virtual assistant, use a translation app, or even get a medical diagnosis, remember that there's a Hydra-like mechanism working behind the scenes, making all these advancements possible.

🛄 Footnotes

Feedback is always welcome! If there are inaccuracies or areas of improvement, please share.
The illustrations were crafted using DALLE, and the content was proofread with the assistance of ChatGPT.

Declarative vs Imperative - Vader Buys a Cake

Odai Athamneh — Sun, 08 May 2022 22:56:37 +0000

Photo by TOMMY VAN KESSEL on Unsplash

There are 2 main programming paradigms:

Declarative. What to do. Think SQL.
Imperative. How to do it. Think C++.

🎂 Vader Buys A Cake

As a memory trick, let's imagine buying a chocolate cake with each approach.

Declarative: You go to the bakery and declare you want one. You don't care how it gets done, you just want the final product.
Imperative: You stand over the baker and tell him exactly what to do. The imperial march plays in the background.

Other Paradigms

Here's how a few other common approaches can be categorized:

📢 Declarative	🎂 Cake
Procedural Programming	Logical Programming
Object Oriented	Functional Programming
Parallel Processing	Database Queries

Static vs Dynamic Typing

Odai Athamneh — Sun, 08 May 2022 19:33:58 +0000

Photo by israel palacio on Unsplash

There are 2 separate concepts to understand:

Strong vs Weak - Can variables change type?
- Strong says no.
- Weak says yes.
Static vs Dynamic - When do we check variable type?
- Static checks at compile time.
- Dynamic checks at runtime.

Examples

	💪 Strong	😖 Weak
🥶 Static	Java, C++	?
👟 Dynamic	Python, Ruby	JavaScript, PHP

Normal vs Applicative Evaluation

Odai Athamneh — Sun, 08 May 2022 18:38:33 +0000

These are 2 types of parameter evaluation. In a nutshell:

Normal = left-to-right
Applicative = inside-out

So for a lambda operation, (double (average 2 4)):

Scheme: eqv? vs equal?

Odai Athamneh — Fri, 06 May 2022 23:36:21 +0000

Photo by Piret Ilver on Unsplash

You can read a few lengthy explanations on Stack Overflow, but in a nutshell:

equal? returns true if its arguments have the same structure. This is comparable to == in most languages.
eqv? is a pointer comparison. It looks at memory addresses.
- The exception is numbers.
- Unlike eq?, it will compare numeric data instead of addresses.
- However, it won't convert between int and float data.

On Concurrency

Odai Athamneh — Fri, 06 May 2022 22:47:50 +0000

Photo by Jørgen Håland on Unsplash

This is a high-level overview of concurrency in programming languages. I'm writing it for my own review purposes, but maybe you'll find it useful too.

Motivations

🤔 Logic. Some problems are naturally concurrent so it makes sense to handle them this way. Think web servers, graphics, AI.
🏎️ Speed. Concurrency lets us fully use multiple processors or cores to maximize computational speed.
📡 Distribution. Machines that work across the internet are inherently concurrent. Writing concurrent software helps to address their unique needs.

Types of Concurrency

There are a few different types of concurrency to be aware of:

Note that coroutines are off to the side. Since they switch between tasks at defined points (not arbitrarily), they aren't truly considered concurrent.

Levels of Concurrency

Going from most atomic (smallest divisions) to least atomic (biggest divisions):

🦐 Instruction-Level. Multiple instructions executing at once. Mostly hardware. Focus of research for 40+ years, but physical limits have been reached.
🪓 Vector-Level. Split up large datasets. Big usage in GPUs. Only benefits certain types of programs.
🐫 Thread-Level. Current focus. Needed for multiple cores. Requires programs to be specially written to take advantage.

What is Scripting?

Odai Athamneh — Fri, 06 May 2022 22:02:21 +0000

Photo by Waldemar Brandt on Unsplash

Scripting is making a robot do boring stuff we don't want to do. 🤖

That might be seen as an overly simple definition, and it probably is. But most definitions I've read online are overly formal and hard to grok.

Examples

👨🏻‍💻 IT guy writes a bash script to update user groups
📊 Manager uses a VBA script to crunch data in Excel
📂 Programmer uses Python to rename a bunch of folders

Comparison

	Scripting	Programming
😤 Difficulty	Easy to learn and use	Challenging
🏭 Compilation	Usually interpreted	Usually compiled
🏎️ Performance	Slower, more memory	Faster, less memory

DEV Community: Odai Athamneh

The Hydra of Machine Learning: Understanding Multi-headed Attention

🚩 TL;DR

📖 Introduction

🐉 Enter the Hydra

🔩 The Nuts and Bolts

🪄 Real-world Applications

🔚 Conclusion

🛄 Footnotes

Declarative vs Imperative - Vader Buys a Cake

🎂 Vader Buys A Cake

Other Paradigms

Static vs Dynamic Typing

Examples

Normal vs Applicative Evaluation

Further Reading

Scheme: eqv? vs equal?

Further Reading

On Concurrency

Motivations

Types of Concurrency

Levels of Concurrency

What is Scripting?

Examples

Comparison