1. Parallel distributed processing (PDP)
Traditional computer program work very differently from brains. That might be why computer programs had been so bad at doing things that brains find easy, such as recognizing objects in picture. To solve this, PDP was developed as an architecture to imitate how the brain works.
According to the book, a PDP system requires eight essential components:
- A set of processing units
- A state of activation
- An output function for each unit
- A pattern of connectivity among units
- A propagation rule
- An activation rule
- A learning rule
- An environment
2. A Brief History of Neural Networks (The two AI winters)
In the past, Marvin Minsky showed that single-layer neural network could not learning critical math function (like XOR gate). For example, a simple computer couldn't correctly classify an object if the input shared characteristics of both categories. In the same book, the authors also show that adding more layers could easily solve this problem. Unfortunately, the public only focused on the negative statement. As a result, funding dried up, and the first AI Winter began.
In the 1980s, researcher started exploring two-layer models. Theoretically, twos layers were enough to learn any mathematical function. However, in practice, these models were too large and too slow to be useful at the time. This triggered the second AI Winter.
Today, technology has finally caught up. We now use GPUs to solve the speed and size limitations of the past. To put in simply: a CPU is great at solving a few big, complex problems, but it takes too long when handling thousands of tiny tasks. On the other hand, a GPU is designed to solve thousands of small problems (like rendering pixels) at the exact same time, By using GPUs to boost performance, AI has finally experienced a massive revival.
3. The Difference between Machine Learning and Deep Learning
Deep learning is just a modern subfield within the boarder discipline of machine learning.
- Machine learning requires humans to manually select and extract features (characteristics) from the data before feeding it to the model.
- Deep learning automatically learns the characteristics of an object directly from raw data without needing human intervention.
4. The Universal Framework: Arthur Samuel's Model
Modern Deep Learning is a direct implementation of Arthur Samuel's concept, but with professional technical terms:
- Architecture: The mathematical structure of the model (previously called Model).
- Parameters: The setting/weights inside the model that change as it learns (previously called Weights).
- Predictions: The model's guesses based on inputs and parameters (previously called Results).
- Loss: The scoring system that measures how wrong the predictions are compared to the Labels (previously called Performance).
5. Universal Approximation Theorem
The Universal Approximation Theorem is a mathematical proof showing that deep neural networks can theoretically solve any complex problem. However, in practice, because of limits in computer hardware and the amount of data we have, we cannot solve literally everything, but it is still incredibly powerful.
6. Current Limitations of Deep Learning
- Requires Labeled Data: Models cannot learn by themselves; humans must label the objects (e.g., tagginf which image is a cat or a dog) before training.
- The "Black Box" Problems: If a model only has a few layers, we can easily see which variables are important. But deep networks have hundreds of layers, making them hard to understand. Fortunately, we can now step-by-step visualize how they work (such as checking what the first layer sees, like lines or shapes).
- Cannot Predict Beyond Training Data: The model will predict poorly if the testing condition is too different from the training data. For example, if we only train it with daytime images, it will fail when testing with nighttime images.
Top comments (0)