DEV Community: Siddhartha Dhar Choudhury

Getting started with PyGeneses

Siddhartha Dhar Choudhury — Sun, 06 Sep 2020 15:12:48 +0000

What is PyGeneses? What is this blog post about? Well if you have no idea what PyGeneses is then I would suggest you to first go through the introductory post. If you know the answer to these two questions, then you are in the right place. Let’s get started.

In this post we will go through installation process and PyGeneses’ training and hyperparameter tuning packages.

Installation

PyGeneses can be installed using pip in either your local system or a cloud based platform. The steps for installation will be the same for both cloud and own system. Let’s install PyGeneses:-

user@programmer:~$ pip install git+https://github.com/Project-DC/pygeneses

It will soon be available in pypi, till then installation will be done from the official GitHub repo for PyGeneses.

Let's start training

pygeneses.envs

Now that we have installed PyGeneses, let’s write some code to train the PrimaVita agents. First, we will look at the most basic code example to train agents with default hyperparameter setting and REINFORCE algorithm (we will be adding more RL algorithms in near future).

Yeah it is just these 3 lines. The three steps to train agents are:-

1) Import PrimaVita (First life) class from envs.prima_vita.

2) Instanstiate the PrimaVita class.

3) Call the run() method on the PrimaVita object.

That was super simple wasn’t it. This default setting will allow you to train the agents in ‘bot’ mode which means you will not be able to see the environment while training, in case you want to see the environment throughout training you just need to change the mode to ‘human’:-

Let us now look at an example where the default hyperparameters are changed:-

You just have to pass a dictionary with keys as the name of hyperparameter (here the initial energy of agents and the time in ticks after which model will be updated are changed) and the values as the value you want to specify. For a more detailed list of available hyperparameters refer to the official docs. Just one more line and you can customize your training.

pygeneses.hypertune

Now let’s move on to hyperparameter tuning package (hypertune) of PyGeneses. PyGeneses has two different algorithms for trying out different hyperparameter values:-

1) Grid Search: This takes in a pool of hyperparameter values as input and trains the model with all the possible combinations of these values. This is useful when you want to try out different values of a hyperparameter or group of hyperparameters.

Following is a snippet of code for applying a grid search over pool of hyperparameters using hypertune:-

This too in 3 lines of code

Again the three steps are as follows:-

1) Import HyperTune class from pygeneses.hypertune.

2) Instantiate HyperTune class with the name of environment class, hyperparameter list, hyperparameter values (2D list) and number of logs to be generated for each run.

3) Finally call hypertuner( ) method of HyperTune class.

The values are to be listed in order in which the hyperparameters are listed.

Moving on to the next algorithm:-

2) Randomized Search: This is similar to grid search in that it too takes a pool of hyperparameter values but instead of training model with all possible combinations it selects a subset out of those combinations randomly based on a probability.

Here is the code example:-

The only change in this code and the grid search code is the extra parameter randomize_percent which tells the randomized search algorithm the percentage of combinations to pick up randomly from the pool of available hyperparameters.

This was the end to this part of PyGeneses blog post series. In the next part we will talk about VitaBoard — an interactive dashboard for visualization of training results in PyGeneses which helps makes sense of the actions taken by agents during training and studying their behaviour. See you there :)

Introduction to PyGeneses

Siddhartha Dhar Choudhury — Sun, 06 Sep 2020 14:58:43 +0000

If I try to summarize PyGeneses in a single line then it would be something like — “PyGeneses is a PyTorch based Deep Reinforcement Learning (Deep RL) framework that helps users to simulate artificial agents in bio-inspired environments”.

Let’s break down the above line:-

PyTorch: PyTorch is a machine learning framework developed by Facebook AI Research (FAIR). It has an intuitive python like syntax which allows faster prototyping and serving of ML/DL models. We use PyTorch for the deep reinforcement learning algorithms that we provide as part of our framework.
Deep Reinforcement Learning: Deep Reinforcement Learning is a technique in Artificial Intelligence that brings together the best of Deep Learning and Reinforcement Learning. Deep learning deals with the concept of neural networks which are capable of learning complex functions from data, and Reinforcement Learning is based on the idea of learning from experience.
Artificial Agents: Artificial agents are simulated and simplified versions of species found in nature. The rules these agents follow closely mimic that of nature which allows them to display behaviour close to what animals around us (or even human beings) display. Note: These agents are nowhere close to the complexity of living beings in our planet and are just a simplified version of them.
Bio-inspired environments: Bio-inspired environments mimic conditions of survival and rules that we see in our natural environment in simulation.

About the framework

PyGeneses is a collection of four different packages:-

pygeneses.envs

This module allows you to create, configure and tweak the in-built bio-inspired environments. As of now, this only provides a single environment called Prima Vita (First Life), but there’s more coming soon! This lets you set up the entire environment and the species in just a few lines of code and provides both high level API and low level control over the environment. Training using the API includes logging of every action of an agent so that it can be studied using VitaBoard.

pygeneses.models
The ‘models’ module is what allows us to import the neural networks which the species uses to learn what to do. As of now, only the default model’s (REINFORCE) implementation is provided, but we will be adding support for custom pluggable networks from v0.2 onwards.

pygeneses.hypertune
The ‘HyperTune’ package allows us to configure and test out various hyperparameters we can provide for an environment and species (a list of hyperparameters is provided in the Classes section of this documentation). This contains single hyperparameter testing, grid search and randomized search. This allows us to find the best set of hyperparameters to display a type of behavior. This also produces logs which we can study using Vitaboard.

pygeneses.vitaboard
Vitaboard provides an advanced, interactive dashboard to study agents after the training phase. After each agent dies, his/her actions are written into a log file. And vitaboard allows us to visualize the agent’s life. It provides us with a life visualizer, group statistics and a genetic history visualizer. It allows us to identify and understand behaviours exhibited by an agent while interacting with the environment or with other agents in the environment.

Use cases

Pygeneses can be used for a variety of purposes, the limit only being your creativity and imagination. That being said, some of the use-cases of pygeneses can be:

1) To understand basic psychology — Since every agent has a limited actions to choose from in each time step, the simulation can be studied to infer the basic psychological patterns displayed, this can help to understand the similarities/dissimilarities between the agents and real life organisms.

2) To create applications/games based on the framework — You can create basic applications/games based on our framework. An example can be a productivity app that adds food to the simulation only when the user completes a task, hence staking the lives of the agents of the user’s own simulation in exchange for productivity.

3) To learn more about Deep Reinforcement Learning — Users can just tinker around with different hyperparameters and models and observe the results visually, this makes for a interesting way to get an insight into the world of Deep Reinforcement Learning.

Target Audience

Due to the simple high level API of PyGeneses I believe that this tool can be helpful for both programmer and non-programmers who want to understand the nature of behaviours in animals or understand concepts of Deep Reinforcement Learning.

Most of our API requires only 3–4 lines of code to modify the environment according to your needs, thus allowing users with little to no knowledge of ML or Deep RL to use our tool easily.

We (the developers of PyGeneses) believe that understanding the reason of display of certain behaviours is not only an active area of study but also a fun experiment and through our framework we want to provide a platform for carrying out both of these tasks.

Final words

If you are someone who is keen to learn about ML and AI, or a scientist who studies behaviour or even someone who just wants to have some fun experimenting with these artificial agents then get your local installation of PyGeneses today.

For a detailed introduction (installation and code examples) follow the subsequent articles on this:-

1) Getting started with PyGeneses

2) Studying logs with VitaBoard

If you want to contribute to this project or simply want to know more, then follow the official docs.