While Deep Learning has made machine learning possible it requires vast data sets that are properly normalized along with vast computing resources ...
For further actions, you may consider blocking this person and/or reporting abuse
Hi! Thank you for a great article.
Unfortunately, I can't point a link now but I recall reading the article which states that while heuristics that require an understanding of a problem bring more sense of scientific satisfaction it's brute-force methods like DL which proved to be effective in a long term run. The reason is as simple as a big rise in computational powers during the decades. And when we are sceptical about DL it's again because we underestimate future rise of computing powers.
And tbh this is quite credible to me.
The question is whether such huge computing resources are required to achieve artificial general intelligence. It appears from simple observation of natural systems that a neural net can be trained without huge data centers and megawatts of power. There is no doubt that back propagation is getting to the same end point, a trained neural network, but it is not a fast learner.
When I talk about learning speed I mean how many observations or experiences it takes to learn something. For example, the unit of experience might be a single game of Go. For a human leaning Go might take perhaps 500 games. For Alpha Zero it took hundreds of millions of games. Because machines operate faster they can play vastly more games and thus have more experience, but their ability to learn from each is minuscule.
My observation is simply that natural systems point toward a better learning solution which once bootstrapped are able to learn from a single experience vs the hundreds or thousands required by current ANNs.
Deep Learning works, but it is like the Model T Ford of cars, or the first aircraft at Kitty Hawk. My article was indicating that many have taken the idea that machine learning requires big data and huge data centers to heart because the only implementations we have to date have these features.
There is a weakness in my appeal to nature, in that often engineered solutions can surpass nature just as modern aircraft vastly surpass birds. But at the same time until we achieve comparable learning performance there is something to be learned by taking inspiration from nature.
You seem to have some fundamental misunderstanding about DL and how it relates to natural neurons.
Neurons were just the inspiration for ANNs, we still don't really know how they work, we do know it's not like an artificial "neuron"1. The human brain is also humongous. A quote that stuck with me from a uni professor: "there are more neurons in the brain than there are stars in the universe". So ANNs differ from brains both in a qualitative and a quantitative way. I'd also argue that ANNs are discrete whereas NNNs are continuous (up to quanta).
Essentially, neural networks are just a way to fit (complex) formulas to training data using gradient descent. Linear regression could be considered the simplest one, albeit it is lacking an activation function, and it draws a straight line through datapoints. I don't think anybody in the field is expecting strong AI from ANNs at this point in time. Maybe it can be combined into something bigger in the future. A better contemporary comparison would be between ANNs and just the visual cortex, though my partner who works in neuroscience would probably get argumentative at that comparison as well.
Finally, I think you are underestimating the learning effort of natural brains. Not only have they been subjected to millions of years of evolution (something we've tried to replicate by evolving neural networks!), we actually do get a fuckton of data in our infancy. Every "frame" of input can be seen as a training sample. Presuming the brain "works at 90 fps", that's over 200 million samples in the first month of life.
You do raise an interesting point about mirroring behaviour, the use of "memes" (by their original definition), in essence. In ANNs, AFAIK the main way to pass knowledge between networks is with pre-training.
A few minor things:
My belief is that the companies and governments who invest in deep learning and have access to these huge datasets, will end up producing generalised models that can be trained on specific tasks with small amounts of data. DL/ML/AI will become anothe readily accessible tool.
I still think of DL as in the very early mobile phone era. It works and everyone can see the benefit, and anyone with money is scrambling to make it useful and be the first to market etc. Once it becomes a product that is affordable/accessible to the average company without having to have millions, then that is when we will see it truly become useful.