DEV Community: Rishal Hurbans

Reinforcement learning with Q-learning

Rishal Hurbans — Wed, 22 Dec 2021 09:27:21 +0000

Like other machine learning algorithms, a reinforcement learning model needs to be trained before it can be used. The training phase centers on exploring the environment and receiving feedback, given specific actions performed in specific circumstances or states.

The life cycle of training a reinforcement learning model is based on the Markov Decision Process, which provides a mathematical framework for modeling decisions. Let's use an autonomous car parking as an example.

A simulator needs to model the environment, the actions of the agent, and the rewards received after each action. A reinforcement learning algorithm will use the simulator to learn through practice by taking actions in the simulated environment and measuring the outcome.

Initialize the environment: This function involves resetting the environment, including the agent, to the starting state. Get the current state of the environment: This function provides the current state of the environment, which will change after each action is performed.

Apply an action to the environment: This function involves having the agent apply an action to the environment - the environment is affected by the action, which may result in a reward.

Calculate the reward of the action: This function is related to applying the action to the environment. The reward for the action and effect on the environment need to be calculated.

Determine whether the goal is achieved: This function determines whether the agent has achieved the goal. In an environment in which the goal cannot be achieved, the simulator needs to signal completion when it deems necessary.

After many cycles of simulation, and the agent taking actions, the Q-table will be trained with sequences of actions that are favourable to get to the destination. Note: Without varying environment data, the model will be overfitted and perform poorly in real-world situations.

Learn more about reinforcement learning in Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Intuition of reinforcement learning

Rishal Hurbans — Wed, 15 Dec 2021 08:02:01 +0000

Reinforcement learning (RL) is an area of machine learning inspired by behavioral psychology. The concept of reinforcement learning is based on cumulative rewards or penalties for the actions that are taken by an agent in a dynamic environment.

Think about a young dog growing up. The dog is the agent in an environment that is our home. When we want the dog to sit, we usually say, “Sit.” The dog doesn’t understand English, so we might nudge it by lightly pushing down on its hindquarters.

After it sits, we pet the dog or give it a treat. This process will need to be repeated several times, and we would have positively reinforced the idea of sitting. The trigger in the environment is saying “Sit”; the behavior learned is sitting; and the reward is treats.

Where supervised learning uses labeled data to make predictions and classifications, and unsupervised learning uses unlabeled data to find clusters and trends, RL uses feedback from actions performed to learn what actions are most beneficial in different scenarios toward a goal.

Introduction to purpose-specific neural networks

Rishal Hurbans — Wed, 08 Dec 2021 09:56:10 +0000

Artificial neural networks (ANNs) are versatile and can be designed to address different problems. Specific architectural styles are useful for solving certain problems. Think of the architecture as being the fundamental configuration of the network.

Convolutional neural networks (CNNs) are designed for image recognition. These networks can be used to find the relationships among different objects and unique areas within images. If you’ve ever searched for an image online, you have likely interacted indirectly with a CNN.

Whereas standard ANNs accept a fixed number of inputs, recurrent neural networks (RNNs) accept a sequence of inputs with no predetermined length. These inputs are like spoken sentences. RNNs are useful in applications pertaining to speech and text recognition and prediction.

A generative adversarial network (GAN) consists of a generator network and a discriminator network. For example, the generator creates a potential solution like an image or a landscape, and a discriminator uses real images of landscapes to determine the realism or correctness.

GANs can be used to generate convincing fake images and videos (also known as deepfakes), which raises concern about the authenticity of information in the media. They can also be used in extraordinary ways for CGI for movies.

Learn more about artificial neural networks in Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Intuition of data in artificial neural networks

Rishal Hurbans — Thu, 02 Dec 2021 05:40:47 +0000

To explore the workings of multi-node ANNs, consider an example dataset related to car collisions. Suppose that we have data from several cars at the moment that an unforeseen object enters the path of their movement.

Speed— The speed at which the car was traveling before encountering the object.
Terrain quality— The quality of the road on which the car was traveling before encountering the object.
Degree of vision— The driver’s degree of vision before the car encountered the object.
Total experience— The total driving experience of the driver of the car.
Collision occurred?— Whether a collision occurred or not.

Because a single hidden node is connected to every input node but every connection has a different weight, independent hidden nodes might be concerned with specific relationships among two or more input nodes.

The nodes in a hidden layer can be conceptually compared with the analogy of ants. Individual ants fulfil small tasks that are seemingly insignificant, but when the ants act as a colony, intelligent behavior emerges.

The nodes of the network all process the inputs through a phase of training to find relationships and understand what relationships influence the classification of the situation.

ANNs are versatile and can be designed to address different problems. Learn more in Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

ML and decision trees for beginners

Rishal Hurbans — Wed, 17 Nov 2021 06:26:44 +0000

Suppose that we have several vehicles that are cars and trucks. We measure the weight of each vehicle and the number of wheels of each vehicle. We also forget for now that cars and trucks look different. How can we make an algorithm tell them apart?

If you haven't already, check out this article for the intuition of machine learning before we continue: https://rhurbans.com/machine-learning-intuition/

Almost all cars have exactly four wheels, and many large trucks have more than four wheels. Trucks are usually heavier than cars, but a large sport-utility vehicle may be as heavy as a small truck. We could find relationships between the weight and number of wheels of vehicles to predict whether a vehicle is a car or a truck. A decision tree can be useful here.

Decision trees are structures that describe a series of decisions that are made to find a solution to a problem. So a decision tree can be used to encompass the above logic. Let's think about how we might train a model to categorize diamonds.

A decision tree includes decision nodes, and leaf nodes. A decision node contains a question being asked, the positive examples related to the question, and the negative examples.

A decision tree also includes leaf nodes containing a list of examples only. These are all examples which were categorised correctly.

In building a decision tree, we test all possible questions to determine which one is the best question to ask at a specific point in the decision tree. To test a question, we use the concept of entropy — the measurement of uncertainty of a dataset.

If we had 5 Perfect diamonds and 5 Okay diamonds, and tried to pick a Perfect diamond by randomly selecting a diamond from the 10, what are the chances that the diamond would be Perfect?

Decision trees are a simple but powerful ML algorithm. To learn more, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Machine learning flavours

Rishal Hurbans — Tue, 09 Nov 2021 10:16:23 +0000

Machine learning is useful only if you have the right data and have questions to ask that it might be able to answer. Machine learning algorithms find patterns in data but cannot do useful things magically.

One of the most common techniques in traditional machine learning is called supervised learning. We want to look at data, understand the patterns and relationships among the data, and predict the results if we are given new examples of different data in the same format.

Regression involves drawing a line through a set of data points to most closely fit the overall shape of the data. Regression can be used for applications such as finding trends between marketing initiatives and sales, or determining rent like in the next example.

A simple example of supervised learning: https://rhurbans.com/machine-learning-intuition/

Classification aims to predict categories of examples based on their features. Can we determine whether something is a car or a truck based on its number of wheels, weight, and top speed?

Unsupervised learning involves finding underlying patterns in data that may be difficult to find by inspecting the data manually. Unsupervised learning is useful for clustering data that have similar features and uncovering features that are important in the relationships.

On an e-commerce site, for example, products might be grouped based on customer purchase behavior. If many customers purchase soap, sponges, and towels together, it is likely that more customers would want that combination of products.

Reinforcement learning is inspired by behavioral psychology and operates by rewarding or punishing an algorithm based on its actions in an environment. It has similarities to supervised learning and unsupervised learning, as well as many differences.

If you're interested in more details about ML algorithms, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Machine learning intuition

Rishal Hurbans — Tue, 02 Nov 2021 11:00:14 +0000

Machine learning can seem like a daunting concept to learn and apply, but with the right framing and understanding of the process and algorithms, it can be interesting, useful, and fun. Let's explore apartment prices in a city.

Suppose that you’re looking for a new apartment. You speak to friends and family, and do some online searches for apartments in the city. You notice that apartments in different areas are priced differently.

A one-bedroom apartment in the city center (close to work) costs $5,000 per month. A two-bedroom apartment in the city center costs $7,000 per month. A one-bedroom apartment outside the city center, where you will need to travel to work, costs $3,000 per month.

You notice some patterns. Apartments in the city center are most expensive and are usually between $5,000 and $7,000 per month. Apartments outside the city are cheaper. Increasing the number of rooms adds between $1,500 and $2,000 per month.

This example shows how we use data to find patterns and make decisions. If you encounter a two-bedroom apartment in the city center with a garage, it’s reasonable to assume that the price would be about $8,000 per month.

Machine learning aims to find patterns in data for useful applications in the real world. We could spot the pattern in this small dataset, but machine learning spots them for us in large, complex datasets.

Intuition of particle swarm optimization

Rishal Hurbans — Tue, 26 Oct 2021 12:35:31 +0000

Swarm intelligence is an amazing phenomena in nature. We see it in flocks of birds, bees in a hive, bacterial growth, and more. The behaviour of these wonderful creatures have been studied and inspired useful algorithms. Here's an introduction to particle swarm optimization.

Particle swarm optimization is a swarm intelligence algorithm. It relies on emergent behavior of many individuals to solve difficult problems as a collective. The most amazing feats are always achieved as a collective. Learn more about swarms here: https://rhurbans.com/optimization-finding-the-best-solutions/

Bird flocks are an ideal example of swarm intelligence in nature. When a single bird is flying, it might attempt several maneuvers and techniques to preserve energy - jumping and gliding or leveraging wind currents to carry it in the right direction of travel.

This behavior indicates some primitive level of intelligence in a single individual. But birds also have the need to migrate during different seasons. In winter, there is less availability of insects and food, nesting locations are scarce, and warm weather is better for survival.

Migration is usually not a short trip. It takes thousands of kilometers of movement to arrive at an area with suitable conditions. When birds travel these long distances, they tend to flock.

The formation that we observe in bird flocks has several advantages. A large, strong bird will take the lead, and when it flaps its wings, it creates uplift for the birds behind it. These birds can fly while using significantly less energy.

Particle swarm optimization involves a group of individuals at different points in the solution space, all using real-life swarm concepts to find an optimal solution. Imagine building a drone and finding the optimal plastic, aluminium, while minimizing drag in the wind.

Each combination of the values for plastic, aliminium, and drag is a possible solution. We want to find the values for plastic and aluminium that minimize the drag value. Particle swarm optimization can help us with this.

PSOs are useful in large search spaces —There are many data points and possibilities of combinations. And search spaces with high dimensions —There are many parameters that influence how well a solution performs.

A PSO particle contains a:

Position —The position of the particle in all dimensions
Best position —The best position found using the fitness function
Velocity —The current velocity of the particle’s movement

Particle swarm optimization provides a means to search a large search space without checking every value in each dimension. These attributes help particles search the solution space and converge to good solutions inspired by swarms in nature.

If you found this interesting and useful, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Learning from ants

Rishal Hurbans — Thu, 21 Oct 2021 10:08:00 +0000

Learning from ants: Ant colony optimization algorithms are versatile and useful for several real-world applications. These applications usually center on complex optimization problems. Here are three uses for the algorithm.

If you haven't already, gain the intuition of the ant colony optimization algorithm here: https://rhurbans.com/ant-colony-optimization-for-beginners/

Route optimization

In a logistics example, perhaps the distance between destinations, traffic conditions, types of packages being delivered, and times of day are important constraints to optimize the operations of the business. ACOs can help with that.

Job scheduling

Job scheduling is required in almost any industry. Nurse shifts are important to ensure that good health care can be provided while keeping nurses healthy. Computational jobs on servers must be scheduled in an optimal manner to maximize the use of the hardware.11

Image processing

Although there are better ways, ACOs can be used for edge detection in image processing. An image is composed of several adjacent pixels, and the ants can move from pixel to pixel, leaving behind pheromone trails that would trace an object.

If you're interested in more details about ant colony optimization and AI, see Grokking AI Algorithms with Manning Publications: [http://bit.ly/gaia-book], consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Ant colony optimization for beginners

Rishal Hurbans — Thu, 09 Sep 2021 11:10:47 +0000

A single ant can carry 10 to 50 times its own body weight and run 700 times its body length per minute. These are impressive qualities; however, when acting in a group, that single ant can accomplish much more.

In groups, ants build colonies, retrieve food, and even use peer pressure to influence others. They achieve this with pheromones - essentially, perfumes that ants drop wherever they go. Other ants can sense these perfumes and change their behavior based on them.

An experiment based on real-life harvesting ants showed that they always converged to the shortest path between the nest and the food source. Ants that have walked specific paths increased the pheromone intensity on those paths that other ants pick up on.

Ant colony optimization (ACO) algorithms simulate the emergent behavior shown in this experiment. In the case of finding the shortest path, the algorithm converges to a similar state, as observed with real ants. How can we use this to solve a more practical problem?

Imagine we're visiting a carnival that has many attractions to experience. Each attraction is located in a different area, with varying distances between them. Because we don’t feel like wasting time walking too much, we want to find the shortest paths between all attractions.

Ant colony optimization can help in finding the shortest path between all the attractions in a more computationally efficient way. If we represent an ant's memory, best distance, and actions it can take, we can simulate a colony and solve hard problems.

Follow for the next part to this thread - @RishalHurbans. To learn the details of this algorithm, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Encoding genetic algorithms

Rishal Hurbans — Wed, 01 Sep 2021 10:57:27 +0000

Genetic algorithms are a fascinating technique for solving optimisation problems. If you can create a set of rules that can measure a solution's performance, you can probably use a GA to help solve the problem. Here are some ways to encode solutions.

If you missed my article on the intuition of genetic algorithms, check this out before continuing: https://rhurbans.com/genetic-algorithms-for-beginners/.

Real-value encoding represents a gene with numeric values, strings, or symbols. The solution is expressed in its natural state. This encoding is used when potential solutions contain continuous values that cannot be encoded easily with binary encoding.

Order encoding, also known as permutation encoding, represents a chromosome as a sequence of elements. Order encoding usually requires all elements to be present in the chromosome. It's useful for problems where the sequence in the solution is important.

Tree encoding represents a chromosome as a tree of elements. Tree encoding is versatile for representing potential solutions where the hierarchy of elements is important or required. Tree encoding can even represent code functions, which consist of a tree of expressions.

Encoding solutions is the first step in setting up a genetic algorithm for success. It can solve interesting problems on its own or even be used with artificial neural networks, like in this Super Mario example.

If you're interested in more details about genetic algorithms, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book, consider following me - @RishalHurbans, or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.

Genetic Algorithms for Beginners

Rishal Hurbans — Fri, 27 Aug 2021 08:23:15 +0000

Genetic algorithms are part of the family of optimization algorithms. They operate on the theory of evolution, more particularly, genetic evolution. Each solution is a chromosome that's made up of genes, and is evaluated to determine how well it performs. This repeats until a good solution is found.

Evolution suggests that the living organisms that we see today did not suddenly exist that way, but evolved through millions of years of subtle changes, with each generation adapting to its environment. See this article for more: https://rhurbans.com/intelligence-through-evolution/.

Genetic algorithms are optimization algorithms. They are used to evaluate massive search spaces for good solutions fast. See this article for more about optimization algorithms: https://rhurbans.com/optimization-finding-the-best-solutions/. Let's dive into how genetic algorithms (GAs) work next.

When we use a GA, it is paramount to do the encoding step correctly. It requires careful design of the representation of possible states. In this case we want to figure out which items to include in the knapsack to maximize the total value and not exceed the weight limit.

Binary encoding represents a gene in terms of 0 or 1, so a chromosome is represented by a string of binary bits. For the knapsack problem, 0 = exclude the item, and 1 = include the item. We can then sum the included items' values to find how well that chromosome performs.

When the search space is small, it's fairly easy for us to figure out a good solution. But what if there were 26 possible items to add to the knapsack? It becomes much more difficult for a human to solve it by hand. Here's where GAs are useful.

A genetic algorithm evaluates the performance of each chromosome and uses mixing to make new solutions that hopefully perform better - like human reproduction. Mixing chromosomes can happen in different ways. A simple approach is to take half and half.

Genetic algorithms also use the concept of mutation. Like in nature, a random gene might be changed after reproduction between two parent chromosomes. This entire process runs for many cycles, called generations, until a good solution is found.

GAs can perform significantly faster than other methods. But remember, they're not guaranteed to find the BEST solution. I'll be elaborating on genetic algorithms with more visual explanations in upcoming articles. Follow @RishalHurbans for updates.

Thank you for reading. If you're keen to learn at your leisure, see Grokking AI Algorithms with Manning Publications: http://bit.ly/gaia-book. Or join my mailing list for infrequent knowledge drops: https://rhurbans.com/subscribe.