When I first came across a mention of Usain Bolt, Leibniz, and Newton together on Khan Academy, it instantly sparked my attention, which makes me want to share it in return.
What could a legendary sprinter and two famous mathematicians have in common? It turns out the answer lies in the basics of calculus, which I realise is more relevant than I initially thought when learning deep learning and artificial intelligence mathematical concepts.
In the context of understanding how neural networks (really work), I logically found myself revisiting the basics of calculus. Oh joy! In this article, I'll share how these three seemingly disparate figures are linked through one obsession that is not only crucial in sports and physics but also, it turns out, in the rapidly evolving field of AI (I know, unbearable suspense!)
Asking a Fundamental Question
Bolt, Leibniz, and Newton were all obsessed with the same fundamental question, one that drives much of differential calculus: What is the instantaneous rate of change of something? This might sound abstract, but it's actually a question that pops up in many real-world situations.
Take Bolt, for example. We might be curious about how fast he’s running at any given moment during a race. Not just his average speed over a second or the next ten seconds, but his exact speed right now. This is precisely what differential calculus helps us figure out—it's all about understanding instantaneous rates of change, the speed of something at a specific moment in time.
At its core, the instantaneous rate of change is all about pinpointing how quickly something is happening at a specific moment. In simpler terms, it’s the speed of a process at one exact point in time. Imagine you're driving a car. While you might be interested in your average speed during a trip, you might also want to know how fast you’re going right now, as you glance at the speedometer. That’s an instantaneous rate of change—the speed of the car at that precise moment.
What Is an Instantaneous Rate of Change?
In mathematics, this concept is crucial because it allows us to move beyond averages and see what's happening in the moment. For athletes like Usain Bolt, it’s the difference between knowing his average speed over a race and understanding his top speed at any given instant. This idea isn't just limited to speed; it applies to any situation where change occurs—whether it’s the temperature outside, the stock market's rise and fall, or even how fast a neural network is learning during training.
Why is this concept so crucial in deep learning?
In mathematics, this concept allows us to move beyond averages and see what's happening in the moment. For athletes like Usain Bolt, it’s the difference between knowing his average speed over a race and understanding his top speed at any given instant. This idea isn't just limited to speed; it applies to any situation where change occurs—whether it’s the temperature outside, the stock market's rise and fall, or, and this is where my interests lie today: how fast a neural network is learning during training.
Calculate the average rate of change over [1, 3] - Khan Academy
What about Derivatives?
You might remember derivatives from your college classes. Personally, I definitely needed a refresher on that fundamental mathematical concept. Instantaneous rates of change and derivatives are deeply connected in calculus. A derivative, in simple terms, measures how a function changes as its input changes. It gives us the rate at which something is happening at any precise moment. So, if we were to graph Usain Bolt's speed over time during a race, the derivative of that graph at any given point would tell us his exact speed at that moment.
Derivatives are powerful because they allow us to move from knowing general trends to understanding specific details in real time.
Read that again.
Derivatives give us the insight needed to see how things are changing right now, not just on average, whether we are calculating Usain Bolt’s speed or fine-tuning the learning speed of a neural network.
Applied derivative example and why it matters in the context of deep learning
Let’s come back to why we arrived here in the first place: understand how neural net really work.
When we dive into the world of deep learning, derivatives become more than just abstract mathematical tools—they are in fact essential for training neural networks. In this context, derivatives are used to optimise the network by adjusting the weights of connections between neurons to minimise errors and improve performance.
Let’s consider how a neural network learns to recognise images.
During training, the network makes predictions and compares them to the actual outcomes. The difference between the prediction and the reality is calculated as an error. To reduce this error, the network uses a process called backpropagation, which relies on derivatives. By calculating the derivative of the error with respect to each weight in the network, we can determine how much each weight should be adjusted to improve the model's accuracy. This adjustment process is repeated many times, gradually tuning the network until it performs well.
In essence, and based on what I learned so far through the Fastai course and Andrej Karpathy’s youtube series, derivatives help guide the network’s learning process, making them indispensable in the development of AI systems. Without understanding and applying these instantaneous rates of change, the sophisticated models that drive advancements in machine learning and artificial intelligence would simply not be possible.
What we learned today:
Instantaneous rates of change and derivatives are crucial concepts that connect theoretical mathematics with real-world applications.
Derivatives allow us to move from understanding general trends to capturing specific, real-time details.
In deep learning, derivatives are essential for optimising and improving neural network performance.
The best takeaway for me is: ##We don’t need super advanced mathematical concepts to learn deep learning!## And that's a really good news.
Keep moving,
Manon 🦉
Top comments (0)