DEV Community: Tim Roderick 🎈

What's Up With Floating Point?

Tim Roderick 🎈 — Thu, 03 Sep 2020 11:23:43 +0000

I presume most people reading this have used floating-point numbers at some point, often not intentionally.

I'm also fairly sure a good number who have encountered them did so after trying to discover why the result of a simple computation is incorrect. E.G.

0.1 + 0.2
// result: 0.30000000000000004
// Which I'm fairly sure
// should be 0.3 ...

The problem is that without understanding what a floating-point number is, you will often find yourself frustrated when the result you expected is ^{ever-so-slightly} different.

My goal here is to clarify what a floating-point number is, why we use them, and how they work.

Why do we even need floating point 🤔?

It's not a bold statement to say that computers need to store numbers. We store these numbers in binary on our phones, laptops, fridges etc.

I hope that most people reading this are familiar with binary numbers in some form; if not, consider reading this blog post by Linda Vivah.

But what about decimal? Fractions, π, real numbers?

For any useful computation, we need computers to be able to represent the following:

Very^{very^{very^very}} SMALL numbers,
Veryveryveryvery BIG numbers,
Everything in-between!

Starting with the very^{very^{very^very}} small numbers to take us in the right direction; how do we store them?

Well, thats simple. We store those using the equivalent to decimal representation in binary...

Binary fractions

An example should help here. Let's choose a random binary fraction: 101011.101

This is very similar to how decimal numbers work. The only difference is that we have base 2 instead of base 10. If you chuck the above in a binary converter of your choice, you'll find that it is correct!

So how might we store these binary fractions?

Let's say we allocate one byte (8 bits) to store our binary fraction: 00000000. We then must choose a place to put our binary separator so that we can have the fractional part of our binary number.

Let's try smack-bang in the middle!

0000.0000

What's the biggest number we can represent with this?

1111.1111 = 15.9375

That's... not very impressive. Neither is the smallest number we can represent, 0.00625.

There is a lot of wasted storage space here alongside a poor range of possible numbers. The issue is that choosing any place to put our point would leave us with either fractional precision or a larger integer range — not both.

If we could just move the fractional point around as we needed, we could get so much more out of our limited storage. If only the point could float around as we needed it, a floating point if you will...

(I'm sorry, I had to 😅.)

So what is floating point?

Floating point is exactly that, a floating ( fractional ) point number that gives us the ability to change the relative size of our number.

So how do we mathematically represent a number in such a way that we,

store the significant digits of the number we want to represent (E.G. the 12 in 0.00000012);
know where to put the fractional point in relation to the significant digits (E.G. all the 0's in 0.00000012)?

To do this, let's time travel (for some) back to secondary school...

Standard Form (Scientific Notation)

Anyone remember mathsisfun? I somehow feel old right now but either way, this is from their website:

You can do the exact same thing with binary! Instead of $7*10^2 = 700$ , we can write

1010111100 * 2^0 = 700 \\ = 10101111 * 2^2 = 700

Which is equivalent to $175 * 4 = 700$ . This is a great way to represent a number based on the significant digits and the placement of the fractional point relative to said digits.

That's it! Floating point is a binary standard-form representation of numbers.

If we want to formalise the representation a little, we need to account for positive and negative numbers. To do this, we also add a sign to our number by multiplying by ±1:

(\text{sign}) * (\text{significant digits}) * (\text{base})^{(\text{some power})}

And back to the example given by mathsisfun...

700 = (1) * (7) * (10)^{2} \\ = (1) * (10101111) * (2)^{2}

If you are reading other literature, you'll find the representation will look something like...

1)^s * c * b^{e}

Where $s$ is the sign bit, $c$ is the significand/mantissa, $b$ is the base, and $e$ is the exponent.

So why am I getting weird errors 😡?

So, we know what floating point is now. But why can't I do something as simple as add two numbers together without the risk of error?

Well the problem lies partially with the computer, but mostly with mathematics itself.

Recurring Fractions

Recurring fractions are an interesting problem in number representation systems.

Let's choose any fraction $xy\frac{x}{y}$ . If $y$ has a prime factor that isn't also a factor of the base, it will be a recurring fraction.

This is why numbers like $1/21$ can't be represented in a finite number of digits; $21$ has $7$ and $3$ as prime factors, neither of which are a factor of $10$ .

Let's work through an example in decimal.

Decimal

Say you want to add the numbers $1/3$ and $2/3$ . We all know the answer is 42 $1$ , but if we are working in decimal, this isn't as obvious.

This is because $1/3 = 0.333333333333...$

It isn't a number that can be represented as a finite number of digits in decimal. As we can't store infinite digits, we store an approximation accurate to 10 places.

The calculation becomes...

0.3333333333 + 0.6666666666 = 0.999999999

Which is definitely not

1

. It's real close, but it's not

1

The finite nature in which we can store numbers doesn't mesh well with the inevitable fact that we can't easily represent all numbers in a finite number of digits.

Binary

This exact same problem occurs in binary, except it's even worse! The reason for this is that $2$ has one less prime factor than $10$ , namely $5$ .

Because of this, recurring fractions happen more commonly in base $2$ .

An example of this is $0.1$ :

In decimal, that's easy. In binary however... 0.00011001100110011..., it's another recurring fraction!

So trying to perform 0.1 + 0.2 becomes

  0.0001100110011
+ 0.0011001100110
= 0.0100110011001
= 0.299926758

Now before we got something similar to 0.30000000004, this is because of things like rounding modes which I won't go into here (but will do so in a future post). The same principle is causing this issue.

This number of errors introduced by this fact are lessened by introducing rounding.

Precision

The other main issue comes in the form of precision.

We only have a certain number of bits dedicated to the significant digits of our floating point number.

Decimal

As an example, consider we have three decimal values to store our significant digits.

If we compare $333$ and $1/3 * 10^3$ , we would find that in our system they are the exact same.

This is because we only have three values of precision to store the significant digits of our number, and that involves truncating the end off of our recurring fraction.

In an extreme example, adding $1$ to $1*10^3$ will result in $1*10^3$ , the number hasn't changed. This is because you need four significant digits to represent $1001$ .

Binary

This exact same issue occurs in binary with very^{very^{very^very}} small and veryveryveryvery big numbers. In a future post I will be talking more about the limits of floating point.

For completeness, consider the previous example in binary, where we now have 3 bits to represent our significant digits.

By using base 2 instead, 1 * 2^3, adding 1 to our number will result in no change. This is because to represent 1001 (now equivalent to 9 in decimal) requires 4 bits, the less significant binary digits are lost in translation.

There is no solution here, the limits of precision are defined by the amount we can store. To get around this, use a larger floating point data type.

E.G. move from a single-precision floating-point number to a double-precision floating-point number.

Summary

TLDR

To bring it all together, floating-point numbers are a representation of binary values akin to standard-form or scientific notation.

It allows us to store BIG and small numbers with precision.

To do this we move the fractional point relative to the significant digits of the number to make that number bigger or smaller at a rate proportional to the base being used.

Most of the errors associated with floating point come in the form of representing recurring fractions in a finite number of bits. Rounding modes help to reduce these errors.

Thanks 🥰!

Thank you so much for reading! I hope this was helpful to those that needed a little refresher on floating point and also to those that are new to this concept.

I will be making one or two updates to this post explaining the in-depths of the IEEE 754-2008 Floating Point Standard, so if you have questions like:

"What are the biggest and smallest numbers I can use in floating point?"
"What do financial institutions do about these errors?
"Can we use other bases?"
"How do we actually perform floating-point arithmetic?"

Then feel free to follow to see an update! You can also follow me on twitter @tim_cb_roderick for updates.If you have any questions please feel free to leave a comment below.

Key Topics from my Computer Science & Maths Degree - Transcript Review

Tim Roderick 🎈 — Thu, 27 Aug 2020 22:52:53 +0000

After four years, I've finally finished my degree 🎉. Since then, I've been thinking about how I developed personally over that time.

This post will aim to highlight the structure and key topics from my degree. Hopefully this will be able to help you decide on some direction to take during your personal development.

I'll sprinkle a few tips throughout 💖.

Already familiar with most of these concepts? Maybe look at a few topics you've dodged until now!

*DISCLAIMER* This is my personal experience, I'm not suggesting this is the correct order or way to learn these concepts. People will learn more/less in a greater breadth/depth, it's personal preference.

First Year

This is how I've described this year to others in the past:

Attack of the concepts 📚.

We were subject to a large number of introductory courses. Taking the time to let this stuff sink in was important long-term.

How to program 👩‍💻

On the CS side, the year mostly consisted of teaching us how to code in C , Java , and Haskell.

C was used to help us understand the lower level aspect of development: how memory is allocated and how pointers work.

After this we learned how to think about programming functionally using Haskell. This felt unnecessary at the time, but it proved incredibly useful for helping me build an intuition for algorithmic complexity.

Finally, Java was used to teach us object oriented programming. Since this is likely the majority of programming a developer will be doing, it is vital to take time to understand OOP principles and design patterns.

If you are looking for a good introductory resource to programming, I love Daniel Shiffman's beginner friendly YouTube channel as a starting resource.

Architecture 👷‍♂️

Introduction to Computer Architecture to me was the most important unit. We aren't programming infinitely complex theoretical boxes as developers, we are using finitely complex machines.

This unit explored computer hardware from silicon to stack.

After learning how to program, this unit helped all the pieces fall into place. It was the first big eureka moment for me as a beginner.

I would highly recommend Prof. Onur Mutlu from ETH Zurich's excellent lecture series on computer architecture.

Necessary Mathematics 🧮

As a developer, you won't get very far if you avoid maths altogether. So embrace it! Also embrace that there is often 100+ years of previous academic research - it's taken people a while to get this stuff.

Look up concepts as and when you need them, taking time to let them sink in. Filling in these spooky-ghost-shaped holes in your knowledge will always help in the long run.

You won't become an expert overnight and mathematics is notorious for its lack of a helping-hand, but you can do it!

Depending on where you want to go with your development, you will need to know different topics. I focused on statistics and machine learning so, for me, the most essential were Linear Algebra and Probability Theory initially.

Notable Mentions 🥉

Theory of computation was a unit dedicated to the fundamental mathematical properties of computing: what can and cannot be computed. Useful IMO. I found these course notes by Vassos Hadzilacos that may be a good read if you're interested.

Second Year

For me, this year was a big motivational bounce-back from the first. It's the move from everything being new and tiring to everything being familiar and challenging. You start to focus in on topics you have knowledge of now and the details start to flow.

My stand-out units from this year all follow this pattern.

Applying New Skills ✨

You know how to write some code ideally by this point (I thought I did 😅). The problem was applying myself to problems that were not designed to educate me.

Data Structures and Algorithms was an important unit all about understanding computational complexity. It was the step from writing code that worked to code that worked efficiently. Dynamic programming was a concept I learned here that I now use regularly. Here's a course PDF I found by John Bullinaria on DSA.

Symbols, Patterns and Signals covered how we digitally interpret the world around us. This was achieved through examples from applications in computer vision, audio, and machine learning.

This topic is broad so some searchable highlights from this unit might be "K-means clustering", "The Fourier Transform", "Audio sampling".

Databases and Cloud Concepts was also taught in my second year, although it was intended to be taught to us in first year. If you tackled this one earlier that would probably make sense. This unit was all about web security, how the internet works, and SQL. Roadmap.sh has a list of resources for learning key concepts with context.

Here's a great video from @3blue1brown about the fourier transform:

Useful Mathematics 🎲

The fundamentals mathematics learnt so far will be enough to keep most developers afloat for the forseeable future. Further reading at this level is still important however.

From the Combinatorics unit I got a much greater understanding of discrete mathematics, including graphs.

Internships 💻

So this is the year I also got my first internship over the summer. If I had to replace the internship, I would self-study what I learnt on the job: C++.

But C++ is one of many programming languages / tools used commonly in industry. Looking into what tools are used in your ideal line of work is a great step.

Third Year

This year was all about taking the conceptual understanding of a variety of topics and churning it into rigorous understanding.

Refining 🔨

These units are all natural extensions of topics learnt from the first two years:

Machine Learning: Linear regression, Gaussian Processes, Unsupervised Learning ...
Image Processing and Computer Vision: Computer Vision algorithms and a bit about Deep Learning.
Web Technologies: How to build a web app; CSS, HTML, Javascript;

I found this blog post that lists several free online courses for ML. For deep learning, fast.ai have practical courses. For web technologies, again Roadmap.sh has a list of resources for learning key concepts with context.

Communicating Science 💬

A skill that is necessary for any developer is your ability to communicate ideas.

Don't have to be a public speaker or a poet.

Don't need to show it in fear of blowing it.

Meek creature; Word weaver.

Chat clearer, not to feature.

Seriously, you do need to be able to communicate your thoughts to your peers and colleagues.

This was the year I did my first serious academic project, which for further reading can be found here. This helped instil some of the key concepts needed for communicating ideas.

As a replacement, this would be a point in your development to start writing blog posts or other long form explanations of concepts. I like this thread I saw on twitter with some ideas of where to start:

Randall Kanna

@randallkanna

A few ideas for your very first dev blog post. 🧵

What are some other good post ideas that you would recommend to a developer?

19:59 PM - 24 Aug 2020

50 292

Fourth Year

From common knowledge to state-of-the-art. The move from understanding topics in reasonable depth to understanding the bleeding-edge.

Specialisation 🎓

The units I took this year describe my specialisation as an academic through a focused set of related subjects.

The following subjects give a good view into my choice of specialisation:

These subjects are related through statistics, machine learning and data science.

At this point in your personal development you may want to find the topics you really enjoy and delve deep into them.

Final Project 🥳

My Masters Thesis was on utilising the popular Transformer architecture for video summarisation. Key point:

It was the most work I've put into anything in my life.
It wasn't successful.
It was fun!

I think this will describe a fair amount of work done at this point in a persons development.

Summary

That covers most of the subjects I took over four years at university! If people want anymore detail on specific subjects then feel free to tweet me @tim_cb_roderick or leave a comment.

Takeaway

Your personal development journey is unique and challenging. By highlighting the steps I took during my education, I hope I can bring clarity and guidance to those looking for it.

The way I learnt certainly isn't the only way.

You may choose to delve deep into certain topics and keep your focus there. You could skip all things hardware and focus purely on the software.

In the end that isn't the key takeaway from this review. For me, it's that learning is a long process that takes time. Rushing it will only hurt your confidence and abilities in the long run.

Resources

I hope I've provided enough resources throughout this post to help guide people if they are interested. If you have any specific queries feel free ask, I'll try my best!

Most of the maths modules I've mentioned so far can be found as free courses on Khan Academy.

A great set of CS courses covering a larger number of topics mentioned in this post can be found at freeCodeCamp.