DEV Community: Edel Prado

Eddie’s Take: Deep Reinforcement Learning with Relational Inductive Biases

Edel Prado — Wed, 09 Feb 2022 16:31:16 +0000

Hello everyone!

I recently came across a conference paper by Zambaldi et al. on deep reinforcement learning with relational inductive biases that I found interesting, but the reading is a bit hard to swallow. With that in mind, I decided to do my take on the paper as a high level overview.

NOTE: I do recommend taking the time to read Deep Reinforcement Learning with Relational Inductive Biases. It explains in detail their entire process of using reinforcement learning. I would highlight the Appendix section. Very cool stuff!

Before we get into the heat of it, let's ask two general questions.

What is reinforcement learning?

Reinforcement Learning (RL) is a method of machine learning where an ‘agent’ learns through interactions with its environment that maximizes the rewards it receives. A good way to think of it would be training a dog using treats as a reward.

What are relational inductive biases?

Relational Inductive Bias places constraints or policies on relationships and interactions among systems in a learning process.

With that out of the way, let’s dive in!

Introduction

The conference paper starts with a quick introduction to recent deep reinforcement learning systems. The positives of recent RL systems are its flexibility in how they learn and utilize the statistical number crunching that is the foundation of our observations and reward signals. Think of the game of pong between a human and a agent using RL. Our agent is being fed frames of the game at hand (observation). If the agent is able to score a point, it will be rewarded positively, and if the human scores a point, the agent will be rewarded negatively. These are the reward signals. By doing this, the agent learns entirely by itself by trying to maximize the rewards.

As for the negatives of RL systems, they include low sample efficiency and poor scaling beyond the set policies of the training environment. What we mean by scaling is that we wouldn’t be able to use the trained RL agent with another game that isn’t pong.

To help counter the negatives, many have turned to using relational inductive biases. The benefits of using relational inductive biases in RL include flexible statistical learning (which helps with the low sample efficiency) and increased structured approaches (helps with scalability).

The authors decided to incorporate relational inductive biases into their RL agent. This led to the RL agent being able to handle challenging tasks. The agent was able to reach state-of-the-art performance on six out of seven StarCraft II mini games. The RL agent even surpassed grandmaster level on those four mini games! On top of that, the authors created what they called a “Box-World” which factored out complex vision spaces. Because of this, the RL agent with relational inductive bias had a higher performance, better efficiency, and is able to generalize an image/frame to solve problems with more complex solutions.

Here is a quick look at the Box-World that is used as input for the RL agent as well as shots of the StarCraft II mini game.

Relational Deep RL Agent Architecture

Here, we will be diving into how the model works. For a visual representation of the agent architecture, see below.

The authors started with a deep RL algorithm based on the A2C method
What is the A2C method? Mike Wang says it best in his published blog on Towards Data Science

In the field of Reinforcement Learning, the Advantage Actor Critic (A2C) algorithm combines two types of Reinforcement Learning algorithms (Policy Based and Value Based) together. Policy Based agents directly learn a policy (a probability distribution of actions) mapping input states to output actions

If it seems a bit complicated, it's because it is! To keep it simple, the process goes something like this.

Input Module - the input is an image of a scene. This would be where our Box-World maps come in handy! The model takes the image and digests it for the relational module to use.
Relational Module - Think of this as the brain of the RL agent. It fires neurons, just like our actual brain, to understand the input and then computes using past data that it collected to find the best course of action.
Output Module - This takes the best course of actions and transforms it to be used in the game.

This is a high level take on the RL agent architecture, so if you'd like to know the math behind it all, I would suggest checking out section 2 of the conference paper.

Experiments and Results

Box-World

This section is all about Box-World! I love the idea of they turned the StarCraft II mini game map into a simplified yet combinatorially complex image. It is made up of a 12x12 pixel room that has keys and boxes randomly scattered. It also has the agent itself in the pixel base room that can move!

To break down how the Box-World works, let talk about how one of the mini game works. There are loose keys (shown as a single colored pixel) that the agent needs to pick up to open boxes (depicted as two colored pixels. One to label it as a box and the second to label the type of key it needs to unlock). Here's the catch! There are multiple boxes and each box has a corresponding key. So the agent will need to understand the key location, move to it, pick it up, move to the corresponding box, open said box, and then collect what is inside. Using the simplified map, it allows the agent to reduce processing a highly detailed image, and generalizes the map which can make it easier to apply in other situations.

Here is Box-World in action!

With the Box-World finished, the team then began experimenting with other StarCraft II mini-games developed for the StarCraft II Learning Environment. The result led to the RL agent to surpass a human grandmaster in for of them!

Here is a performance chart on this.

This, in my opinion, is incredible!

The paper then continues with more details into how it all works as well as applicability to other tasks.

Ending Note: All the images came from the conference paper (the topic of this blog) which be found here. It is a well thought out and informative paper that I'm glad I decided to read.

BONUS: Check out this video of DeepMind Starcraft going up against Grzegorz 'Mana' Komincz who plays on Team Liquid esports.

Welcome to Docker! 🐳

Edel Prado — Tue, 21 Dec 2021 05:31:44 +0000

Hi! I’m Eddie Prado and I wanted to showcase the amazing Docker platform! In this blog, I want to explain to you what docker is, how it works, and reasons why you should learn the ins and outs of it.
_

What is Docker?

First things first, what is Docker exactly? To put it simply, it is a platform for building, running, and shipping applications. So, if the app you are developing works on your machine, it will run on someone else’s machine via containers.
_

Let’s look at an example where Docker would have been a great tool to have:

Developer A is part of a team that are creating an app that could revolutionize how we see data. Dev-A, after long sleepless nights, gets his code running! Woohoo! Now he needs to send it to Developer B for testing and review. Dev-B downloads Dev-A’s app onto to his machine, but it won’t run. After half a workday of trial and error, Dev-A and Dev-B have the app working by updating, downgrading, and replacing software and files. Phew! Now Dev-B can finally do his testing, but the app still must go through many other channels before it can publish. This scenario will keep repeating itself unless the team matches Dev-A’s machine configuration to their own machine which is just not possible when everyone is working on their own distinct projects that require different configurations. So, how could we solve this problem?

Well, there are two routes you could possibly go.
_
_

🖥 Virtual Machines Vs. Docker! 🐳

One way to resolve this issue would be to use a virtual machine. A virtual machine, or VM for short, isn’t any different than other physical computers. It uses its own CPU, memory, disk, and can even connect to the internet! While the hardware of a computer is physical and tangible, VMs are often thought of as software-defined computers within physical servers. While virtual machines may have some positives like trying new operating systems, running old software, or developing software for other systems, the negatives severely slow down the development process.

One bad side of VMs is the need to have a operating system for every virtual machine you use. On top of that, those operating systems need to be licensed, patched, and monitored. Another downside would be the slow start up. Just like physical computers, VMs need to boot up, and depending on the task at hand, you may need to boot up several different virtual machines. This leads me to my final point. Virtual machines are resource intensive. These resources being CPU, memory, and disk space allocation.
What could counter these negatives? Docker, that’s what!

Docker turns the negatives from VMs into positives, and by doing so, it streamlines the app development process! While virtual machines need their own operating systems, Docker does not. It uses the hosts own OS kernel, so there is no need to buy a separate license or worry about extra patches. Since this eliminates the need of a guest OS which is essential for virtual machines to function, Docker is incredibly lightweight. Because of this, Docker boot up times are minimal when compared to VMs, and Docker uses less resources by downloading only the most essential files/software to run an application.

Here is a comparison of the infrastructure between Docker and Virtual Machines

_
_

How does Docker work?

To answer that question, there are three things you must know.

Dockerfile - which is plan for building a Docker image.
Image - which is a template for running a container
Container - which is a running process/application

The process starts with replicating an environment (like Dev-A’s machine configuration!). The person who is developing the application creates the environment using the dockerfile. After the dockerfile has been created, any developer will have the ability to create an image, which is an immutable snapshot of the environment, using the dockerfile. These images can be pushed up to the cloud (DockerHub) for others to pull down to their local machine. They then can create a container which is a running process of the image that was pulled from the cloud.
_

Let’s go back to our Dev-A and Dev-B example. This time we will use Docker!

Dev-A is in charge of creating an app that could revolutionize how we use data. Before they start their long journey of developing said application, they create a dockerfile with a base image in the root of the project. Dev-A will continue to update it with the necessary files used in the app throughout his process. An unspecified amount of time later, Dev-A finishes his app! They then build an image, which creates an image ID, using the terminal. Dev-A then pushes said image to DockerHub and gives the image key to Dev-B to pull the image down to his local machine. Dev-B does just that and begins running an isolated container which flawlessly runs the app. Developer B instantly gets to work testing and reviewing the code while Dev-A goes home and gets his much needed sleep.

When working in a team, Docker is essential in creating a efficient and streamlined development process. It not only increases productivity, but also allows for multiple containers to run on the same machine, maintain isolated applications, and allows for quick and easy configurations!
_
_

Where to download the Docker Desktop application?

To install Docker head to https://docs.docker.com/get-docker/ and download the Docker Desktop application for your operating system. Docker is available for Mac, Windows, and Linux. There is also a VS Code extension for docker. Check out Tutorial: Get started with Docker to get more details on that.

I heavily recommend checking out the below resources to get an even better grasp on how powerful Docker is. Especially when collaborating as a team.
_

Resources:

Video:
Great video explaining how Docker works.

_

Extra Reading:

Ready, Set, Data!

Edel Prado — Tue, 30 Nov 2021 13:54:08 +0000

Hello!

My name is Edel (Eddie) Prado. I am currently on a journey to becoming an effective data scientist at Flatiron School! Flatiron is a bootcamp where they jam all the knowledge you need to be successful in the career path you choose. In my case, data science. So far in my journey, it’s been a rollercoaster of emotions and time management. The curriculum is fast paced and incredibly hands on which is a plus for me since I’m always up for a challenge of sorts. What led me to pursue data science? What do I hope to achieve? Exactly who is Eddie? These questions are stories in themselves.

Chasing After Data

To understand my reasoning behind the decision to pursue data science, I will take a bit of time to explain my background. I was an Operations and Quality Analyst at a steel service center for about four years. The title itself, in my opinion, did not represent my responsibilities within the company very well. My responsibilities included handling production part approval processes (PPAPs), internal audits, safety audits, layered processing audits (LPAs), and the like. It was a good workload, but I always felt the need to do more. Because of this feeling, I started to track different kinds of information that was readily available to me. This included safety incidents, machine downtime/repairs, and customer satisfaction. After teaching myself a bit of VBA coding, I ended up with a ton of data, but I didn't know what to do with it. I started researching ways to help simplify and easily order the information. That's when I first found the term data science and the sampling of the tools data scientist use to turn chaotic data into ordered and informative visualizations. I was starting to enjoy myself the more I dove deeper into the inner workings of a data scientist. So much so that I started a career change to become one through Flatiron School!

My Future in Data

What exactly am I hoping to achieve? How will I accomplish them? Those are questions I consistently ask myself and to my surprise, I do have an answer. Albeit a somewhat vague and everchanging answer but an answer nonetheless!

My top priority is to finish my 5-month bootcamp through Flatiron. Which is honestly the most based goal I have. I just finished one week of the bootcamp, and I already exhausted all my previous experience in coding on day two! So, all the lessons, labs, and quizzes are challenging for me. There are times where I can feel myself going bald from the sheer frustration of not getting code to work the way I want. But when it does! Oh boy, does that sense of accomplishment feel good. That short state of bliss is what makes me go “Okay, now how could I make this better?” This usually ends up with me losing complete track of time though.

My next goal on the list would be expanding my knowledge of different subjects. I love seeing statistics. Topics like the environment, schooling, gaming, and development are some of the things that I care most about. Me becoming data scientist will lead me to understanding these subjects by gathering data, finding correlations, and presenting my findings (after extensive cleaning and engineering) to those who care. This is what I find the most exciting about data science!

Another goal on the list would have to be playing with data in general. For context, I absolutely love computer gaming. Let’s pump the brakes for a sec. Why would I go for data science over any other gaming focused professions? Well, the games I play are massively multiplayer online games (MMOs). These include World of Warcraft (WoW), Final Fantasy XIV (FFXIV), and Albion (which I just started recently). These games all have massive amount of customization and functions within them. This leads to a good chunk of data to play with called logs. The need to become the number one player in a game of millions led to many gamers to data science without even knowing. The proof would be the creation of a website called Warcraft Logs. This website takes all the combat data (usually a txt file) from a fight or raid and converts it to comprehensible visualizations. The most amazing thing is that there are even more sites that take all that data and advise you on how to improve from pressing specific abilities in a certain order to advising on what gear to equip your character with. I have loads of experience with these websites and I didn’t know that I was doing a type data science. I just wanted to be a better player for my group. After a while, I started taking interest on how to get my own charts through data, so I started to save logs in the hope that I could try my hand on cleaning, exploring, and visualizing them. With all that said, I have a goal to play with this data and come out with helpful and informative output that will make me a better player.

The Gaming Data Fanatic Dad

I am a father of three. A son who is turning four in December and twin girls who just recently turned three. I love my kids and wife, and they are the reason to why I jumped into the opportunity to elevate my career through data science. If you couldn’t tell already, I have a passion for video games. Not particularly on the development side, but the information available to players in game and as well as out. I hope to join a company in this setting, but I will follow the data to where it leads me.

All in all, I am incredibly thankful for all the support from my family and friends, and I plan to show them the results of my journey to becoming an effective data scientist!

Keep trekking,
-Eddie P.