DEV Community: Darren Broderick (DBro)

Unleashing the Future: AWS-sponsored STEAM Event Ignites Passion for Machine Learning

Darren Broderick (DBro) — Wed, 13 Mar 2024 16:03:10 +0000

On January 23rd, 2024, the Mid Antrim Museum in Ballymena hosted an inspiring AWS-sponsored event, immersing students in robotics and machine learning. Physicist Brian Cox inspired four groups to explore the captivating world of machine learning, focusing on the AWS DeepRacer League—an intersection of technology and education.

A key highlight for me was to demystify machine learning and mathematics for the students attending the track, using the AWS DeepRacer kit to demonstrate real-world applications in autonomous racing.

The event’s comprehensive curriculum provided students with a holistic understanding of machine learning and robotics. They engaged with the AWS DeepRacer kit, designing and racing models on the track, bridging theory and application.

Professor Brian Cox, renowned for science education, enthusiastically joined the DeepRacer fun, winning second place in a friendly competition.

I had a fantastic day representing OpenData Belfast and AWS by teaching students more about opportunities in data and machine learning. On the track we had a lot of engagement throughout the day with a personal highlight of having Professor Brian Cox getting hands on and even asking a few questions about ODB!”

The Mid Antrim Museum transformed into an innovation hub, underscoring the connection between technology and education. This AWS-supported event delivered an exhilarating day of learning and discovery, spotlighting the dedicated support for STEAM education!

2024 : AWS DeepRacer Local Training On DRFC

Darren Broderick (DBro) — Wed, 13 Mar 2024 14:29:50 +0000

Training Locally on DRFC ->Troubleshooting

Handy Commands
dr-upload-model -b -f -i
dr-upload-model -b -f -I "name of model"
If you have disk problems do "docker system prune"

So you’re training locally on DRFC for AWS DeepRacer, great!

But it’s not always straightforward, sometimes it’s easy to forget how to run or update your stack after the inital setup or you get new errors when starting training again, especially after a season break.

This article an be used as a supplement to the main DRFC guide.
https://aws-deepracer-community.github.io/deepracer-for-cloud

It’s a list of commands/ steps I follow and troubleshooting problems / solutions I’ve faced when training locally.

Hopefully it can help you too, BUT it is tailored to how I run things FYI.

Contents

Handy monthly items
General Training Starting Steps
Virtual DRFC Upload
Physical DRFC Upload
Container Update Links
Open GL Robomaker
New Sagemaker -> M40 Tagging
Log Analysis
Run Second DRFC Instance
Steps for fresh DRFC
Troubleshooting DRFC (List of issues & solutions)
Miscellaneous

Handy monthly items
Latest Robomaker Container (For Training)
https://hub.docker.com/r/aws deep racercommunity/deepracer-robomaker/tags?page=1&ordering=last_updated

All Track Files & Details (For DR_WORLD_NAME & Log Analysis)
https://github.com/aws-deepracer-community/deepracer-race-data/tree/main/raw_data/tracks

Commands

docker ps -a
docker images
docker service ls

General Training Starting Steps
These are commands I run if starting from a reboot

source bin/activate.sh
sudo liquidctl set fan1 speed 30
(This is my own fan setting)
dr-increment-training -f
dr-update OR dr-update-env (I tend to favour -env)
dr-start-training OR dr-start-training -w
dr-start-viewer OR dr-update-viewer
http://127.0.0.1:8100 OR http://localhost:8100
dr-logs-robomaker (dr-logs-robomaker -n2) for worker 2 etc
dr-logs-sagemaker
nvidia-smi (check temperatures)
htop to check threads and memory usage
(Try to maximise my worker count, but keep to <75%)
dr-start-evaluation -c & dr-stop-evaluation
Virtual DRFC Upload
aws configure
dr-upload-model -b -f
Uploads best checkpoint to s3
Physical DRFC Upload
dr-upload-car-zip -f
Sagemaker must be running for this to work
Only uses last checkpoint, not best
Container Update Links
Check your version with command ”docker images”

docker service ls to make sure you see s3_minio.

Sagemaker
https://hub.docker.com/r/awsdeepracercommunity/deepracer-sagemaker/tags?page=1&ordering=last_updated
For new Sagemaker images follow this guide:
https://github.com/aws-deepracer-community/deepracer-for-cloud/blob/master/docs/multi_gpu.md
Robomaker
https://hub.docker.com/r/aws deep racercommunity/deepracer-robomaker/tags?page=1&ordering=last_updated
RL Coach
https://hub.docker.com/r/awsdeepracercommunity/deepracer-rlcoach/tags
Linux terminal startup script is called “.bashrc”

Open GL Robomaker
https://aws-deepracer-community.github.io/deepracer-for-cloud/opengl.html

example -> docker pull awsdeepracercommunity/deepracer-robomaker:4.0.12-gpu-gl
system.env: (Below bullet points)
DR_HOST_X=True; uses the local X server rather than starting one within the docker container.
DR_ROBOMAKER_IMAGE; choose the tag for an OpenGL enabled image - e.g. cpu-gl-avx for an image where Tensorflow will use CPU orgpu-glor an image where also Tensorflow will use the GPU.
Do echo $DISPLAY and see what that is, should be :0 but might be :1
Make system.env dr_display value same as echo value
dr-reload
source utils/setup-xorg.sh

source utils/start-xorg.sh
you should see the xorg stuff in nvidia-smi once you run the start-xorg.sh script
sudo pkill x11vnc
sudo pkill Xorg

New Sagemaker — M40 Tagging (redunant from v5.1.1)
With the latest images you don’t need to compile a specific image (like your -m40 image)

run -> docker tag 2b4e84b8c10a awsdeepracercommunity/deepracer-sagemaker:gpu-m40

Log Analysis

run -> dr-start-loganalysis
Only change needed is for model_logs_root
e.g. ‘minio/bucket/model-name/0’
All Track files & details
https://github.com/aws-deepracer-community/deepracer-race-data/tree/main/raw_data/tracks
Might have to upload the new track to tracks folder
Repo for all racer data
https://github.com/aws-deepracer-community/deepracer-race-data/tree/main/raw_data/leaderboards
Run Second DRFC Instance
Create 2 different run.env or use 2 folders
The DR_RUN_ID keeps things separate
Only 1 minio should be running
Use a unique model name
Run source bin/activate.sh run-1.env to activate a separate environment
Steps for fresh DRFC
https://aws-deepracer-community.github.io/deepracer-for-cloud/installation.html
./bin/prepare.sh && sudo reboot
docker start
ARCH=gpu
Run LARS script -> source bin/lars_one.sh
docker swarm init (If issues run step 7 and grab IP, run step 8, check bottom for example)
ifconfig -a
docker swarm init
docker swarm init — advertise-addr 000.000.0.000
sudo ./bin/init.sh -a gpu -c local
docker images
docker tag xxxxxxx awsdeepracercommunity/deepracer-sagemaker:gpu-m40
source bin/activate.sh
vim run.env
vim system.env
dr-update
aws configure — profile minio
aws configure
(use real AWS IAM details below to allow upload of models)
dr-reload
docker ps -a
Setup multiple GPU
cd custom-files
vim on the 3 files
dr-upload-custom-files
Different editor option to vim

gedit

Troubleshooting DRFC (List of issues & solutions)

General Tip

It’s always worth checking if you are missing anything new that might have been added to the default files that DRFC would then be expecting.

In particular, the system.env or template-run.env files and compare them with your own.

Troubleshooting Docker Start

Docker failed to start

docker ps -a
docker service ls
sudo service docker status
sudo service — status-all
sudo systemctl status docker.service
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

sudo systemctl stop docker
sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl restart docker
sudo service docker restart
snap list
sudo su THEN apt-get install docker.io
Re-run Installing Docker (From Lars)
cat /etc/docker/daemon.json
apt-cache policy docker-ce
sudo tail /var/log/syslog
sudo cat /var/log/syslog | grep dockerd | tail
“For me it was a missing file”

sudo gedit /etc/docker/daemon.json
Make /etc/docker/daemon.json look like below:
{
“runtimes”: {
“nvidia”: {
“path”: “nvidia-container-runtime”,
“runtimeArgs”: []
}
},
“default-runtime”: “nvidia”
}

Make /etc/docker/daemon.json look like below:
sudo systemctl stop docker then sudo systemctl start docker
test with -> docker images
Troubleshooting Docker Swarm

Could not connect to the endpoint URL: “http://localhost:9000/bucket

Error response from daemon: This node is not a swarm manager. Use “docker swarm init” or “docker swarm join” to connect this node to swarm and try again.

You might have to disable ipv6 to stop docker pulling from multiple addresses

Here’s how to disable IPv6 on Linux if you’re running a Red Hat-based system:

Open the terminal window.
Change to the root user.
Type these commands:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
To re-enable IPv6, type these commands:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=0
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=0
sysctl -p
run -> ./bin/init.sh (Resets run, system.env, hyperparam, RF & model_metadata)

run -> docker pull minio/minio:RELEASE.2022–10–24T18–35–07Z

DR_MINIO_IMAGE in system.env, make sure it’s set to:
RELEASE.2022–10–24T18–35–07Z

Useful Links

Full Guide — https://aws-deepracer-community.github.io/deepracer-for-cloud
Sudo — https://phpraxis.wordpress.com/2016/09/27/enable-sudo-without-password-in-ubuntudebian
Training on multiple GPU — https://github.com/aws-deepracer-community/deepracer-for-cloud/blob/master/docs/multi_gpu.md
nvidia monitor — https://stackoverflow.com/questions/8223811/a-top-like-utility-for-monitoring-cuda-activity-on-a-gpu
Tesla M40 24GB specs — https://www.microway.com/hpc-tech-tips/nvidia-tesla-m40-24gb-gpu-accelerator-maxwell-gm200-close
Complex shutdown — https://www.maketecheasier.com/schedule-ubuntu-shutdown
Sudo shutdown — https://sdet.ro/blog/shutdown-ubuntu-with-timer
Video trimmer — https://launchpad.net/~kdenlive/+archive/ubuntu/kdenlive-stable
Flatpak — https://flatpak.org/setup/Ubuntu

Installation commands

sudo snap install jupyter
sudo apt install git
sudo apt install nvidia-cuda-toolkit
sudo apt install curl
sudo apt install jq
sudo pip install liquidctl (to install fan controller globally)
sudo apt install net-tools
sudo apt install vim
sudo apt-get install htop
sudo apt install hddtemp
sudo apt install lm-sensors
pip install — user pipenv
sudo apt install pipenv
pipenv install jupyterlab

Installing Docker

sudo su (run from root)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable”
sudo apt-get update && sudo apt-get install -y — no-install-recommends docker-ce docker-ce-cli containerd.io
sudo apt-get install -y — no-install-recommends nvidia-docker2 nvidia-container-toolkit nvidia-container-runtime
sudo apt-get upgrade

Steps for Cuda upgrade

First removed existing:
sudo dpkg -P $(dpkg -l | grep nvidia-driver | awk ‘{print $2}’)
sudo apt autoremove
then added new:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv — fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository “deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /”
sudo apt update
sudo apt -y install cuda
then rebooted and do nvidia-smi

DeepRacer Virtual Racing

Darren Broderick (DBro) — Thu, 23 Feb 2023 16:12:02 +0000

Latest Track File(For Log Analysis)
https://github.com/aws-deepracer-community/deepracer-simapp/tree/master/bundle/deepracer_simulation_environment/share/deepracer_simulation_environment/routes

Latest Robomaker Container (For Training)
https://hub.docker.com/r/aws deep racercommunity/deepracer-robomaker/tags?page=1&ordering=last_updated

General Training Starting Steps

These are commands I run if starting from a reboot
source bin/activate.sh
sudo liquidctl set fan1 speed 30
(This is my own fan setting)
dr-increment-training -f
dr-update OR dr-update-env (I tend to favour -env)
dr-start-training OR dr-start-training -w
dr-start-viewer OR dr-update-viewer
http://127.0.0.1:8100 OR http://localhost:8100
dr-logs-robomaker (dr-logs-robomaker -n2) for worker 2 etc
dr-logs-sagemaker
nvidia-smi (check temperatures)
htop to check threads and memory usage
(Try to maximise my worker count, but keep to <75%)
dr-start-evaluation -c & dr-stop-evaluation

Virtual DRFC Upload

aws configure
dr-upload-model -b -f
Uploads best checkpoint to s3

Physical DRFC Upload

dr-upload-car-zip -f
Sagemaker must be running for this to work
Only uses last checkpoint, not best

Container Update Links

Check your version with command "docker images"

Sagemaker
https://hub.docker.com/r/awsdeepracercommunity/deepracer-sagemaker/tags?page=1&ordering=last_updated

For new Sagemaker images follow this guide:
https://github.com/aws-deepracer-community/deepracer-for-cloud/blob/master/docs/multi_gpu.md

Robomaker
https://hub.docker.com/r/aws deep racercommunity/deepracer-robomaker/tags?page=1&ordering=last_updated

RL Coach
https://hub.docker.com/r/awsdeepracercommunity/deepracer-rlcoach/tags
Linux terminal startup script is called ".bashrc"

Open GL Robomaker

https://aws-deepracer-community.github.io/deepracer-for-cloud/opengl.html
example -> docker pull awsdeepracercommunity/deepracer-robomaker:4.0.12-gpu-gl
system.env: (Below bullet points)

DR_HOST_X=True; uses the local X server rather than starting one within the docker container.

DR_ROBOMAKER_IMAGE; choose the tag for an OpenGL enabled image - e.g. cpu-gl-avx for an image where Tensorflow will use CPU orgpu-glor an image where also Tensorflow will use the GPU.
Do echo $DISPLAY and see what that is, should be :0 but might be :1
Make system.env dr_display value same as echo value
dr-reload

source utils/setup-xorg.sh
source utils/start-xorg.sh
you should see the xorg stuff in nvidia-smi once you run the start-xorg.sh script
sudo pkill x11vnc
sudo pkill Xorg

New Sagemaker - M40 Tagging

run -> docker tag 2b4e84b8c10a awsdeepracercommunity/deepracer-sagemaker:gpu-m40

Log Analysis

run -> dr-start-loganalysis
Only change needed is for model_logs_root
e.g. 'minio/bucket/model-name/0'
Tracks
https://github.com/aws-deepracer-community/deepracer-simapp/tree/master/bundle/deepracer_simulation_environment/share/deepracer_simulation_environment/routes
Might have to upload the new track to tracks folder
Repo for all racer data
https://github.com/aws-deepracer-community/deepracer-race-data/tree/main/raw_data/leaderboards

Run Second DRFC Instance

Create 2 different run.env or use 2 folders
The DR_RUN_ID keeps things separate
Only 1 minio should be running
Use a unique model name
Run source bin/activate.sh run-1.env to activate a separate environment

Steps for fresh DRFC

./bin/prepare.sh && sudo reboot
docker start
ARCH=gpu
Run LARS script -> source bin/lars_one.sh
docker swarm init (If issues run step 7 and grab IP, run step 8, check bottom for example)
ifconfig -a
docker swarm init
docker swarm init - advertise-addr 000.000.0.000
sudo ./bin/init.sh -a gpu -c local
docker images
docker tag xxxxxxx awsdeepracercommunity/deepracer-sagemaker:gpu-m40
source bin/activate.sh
vim run.env
vim system.env
dr-update
aws configure - profile minio
aws configure
(use real AWS IAM details below to allow upload of models)
dr-reload
docker ps -a
Setup multiple GPU
cd custom-files
vim on the 3 files
dr-upload-custom-files

Different editor option to vim
gedit

Troubleshooting DRFC

General Tip
It's always worth checking if you are missing anything new that might have been added to the default files that DRFC would then be expecting.
In particular, the system.env or template-run.env files and compare them with your own.

Troubleshooting Docker Start

Docker failed to start
docker ps -a
docker service ls
sudo service docker status
sudo service - status-all
sudo systemctl status docker.service

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
sudo systemctl stop docker
sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl restart docker
sudo service docker restart
snap list
sudo su THEN apt-get install docker.io
Re-run Installing Docker (From Lars)
cat /etc/docker/daemon.json
apt-cache policy docker-ce
sudo tail /var/log/syslog
sudo cat /var/log/syslog | grep dockerd | tail

"For me it was a missing file"
udo gedit /etc/docker/daemon.json
Make /etc/docker/daemon.json look like below:

{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia"
}
Make /etc/docker/daemon.json look like below:
sudo systemctl stop docker then sudo systemctl start docker
test with -> docker images

Troubleshooting Docker Swarm

Could not connect to the endpoint URL: "http://localhost:9000/bucket
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
You might have to disable ipv6 to stop docker pulling from multiple addresses
Here's how to disable IPv6 on Linux if you're running a Red Hat-based system:
Open the terminal window.
Change to the root user.
Type these commands:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.tun0.disable_ipv6=1
To re-enable IPv6, type these commands:
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
sysctl -w net.ipv6.conf.tun0.disable_ipv6=0
sysctl -p

run -> ./bin/init.sh
run -> docker pull minio/minio:RELEASE.2022–10–24T18–35–07Z
DR_MINIO_IMAGE in system.env, make sure it's set to:
RELEASE.2022–10–24T18–35–07Z

Other Fixes That Might Work for minio

run -> docker swarm init
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
run -> docker swarm leave
run -> docker swarm init
Error response from daemon: could not choose an IP address to advertise since this system has multiple addresses on interface
run -> docker network ls
sagemaker-local should appear in the network
IF NOT
There's a new fix script for this called "lars_swarm_fix.sh" in the bin folder.
run-> docker swarm leave - force
run -> source bin/lars_swarm_fix.sh

Script might need address, error message will say, This node is not a swarm manager. Use "docker swarm init"
run -> docker swarm init (and grab the first addr, example below)
docker swarm init - advertise-addr 2a00:23c8::d6c3:4a71:9adb:87ad

Swarm initialized: current node (wv3eqpslrstc6hm7n65744z) is now a manager.
ifconfig -a
You don't need to join the token
dr-start-training

Swarm is a docker concept, you can theoretically connect multiple machines together and run DRFC over multiple machines, sagemaker on one PC, robomakers spread out, but once you have cloned DRFC you can now do bin/init.sh -a gpu -c local

Issue - Minio kept making new containers every 10 seconds

Fix: https://github.com/aws-deepracer-community/deepracer-for-cloud/pull/102/commits/a2db4df0a624ace87b89afcc7ff27f35fe9751fe
docker service rm s3_minio
source bin/activate.sh

Issue - Minio containers kept exiting within 7 seconds
docker ps -a
docker service rm s3_minio
docker-compose -f $DR_DIR/docker/docker-compose-local.yml -p s3 up
docker ps
ls -l data
ls -l
Issue was I ran the init script as root
Fix -> chown -R dbro:dbro .
docker-compose -f $DR_DIR/docker/docker-compose-local.yml -p s3 up
docker ps -a showed there were now 2 minio's running
docker-compose -f $DR_DIR/docker/docker-compose-local.yml -p s3 down
docker stack rm s3
dr-reload
docker ps
dr-upload-custom-files

- - - - - General Notes - - - - -

The m40 runs sagemaker docker
System ram runs robomaker
You can offload some of the robomaker to gpu by using the opengl image, but generally yes
Basically model is living inside the GPU memory
training checkpoints are in - cd data/minio/bucket

Wouldn't go any higher than what "htop" shows below because you're at 80% on all threads
- - - - - Additional Scripts - - -

Create script "lars_one.sh"

if [[ "${ARCH}" == "gpu" ]];
then
    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
    curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
    curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
    sudo apt-get update && sudo apt-get install -y --no-install-recommends nvidia-docker2 nvidia-container-toolkit nvidia-container-runtime
    cat /etc/docker/daemon.json | jq 'del(."default-runtime") + {"default-runtime": "nvidia"}' | sudo tee /etc/docker/daemon.json
fi

Miscellaneous

Sensors

"FRONT_FACING_CAMERA"
"SECTOR_LIDAR"
"LIDAR"
"STEREO_CAMERAS"

Check temperature commands

nvidia-smi
nvidia-smi -l 60
watch -n900 nvidia-smi (Every 15 minutes auto calls)
sensors

Set fan speed commands

sudo liquidctl set fan1 speed 30
sudo liquidctl set fan1 speed 0

Check specs / stats commands

nvidia-smi -L
GeForce GTX 1650 -> nvidia-smi -a -i 0
M40 Specs -> nvidia-smi -a -i 1
lspci -k | grep -EA3 'VGA|3D|Display'
top (checks processors to help see worker limits)
free -m
htop
docker stats
docker run - rm - gpus all nvidia/cuda:11.6.0-base-ubuntu20.04 nvidia-smi

Useful Links

Full Guide - https://aws-deepracer-community.github.io/deepracer-for-cloud
Sudo - https://phpraxis.wordpress.com/2016/09/27/enable-sudo-without-password-in-ubuntudebian
Training on multiple GPU - https://github.com/aws-deepracer-community/deepracer-for-cloud/blob/master/docs/multi_gpu.md
nvidia monitor - https://stackoverflow.com/questions/8223811/a-top-like-utility-for-monitoring-cuda-activity-on-a-gpu
Tesla M40 24GB specs - https://www.microway.com/hpc-tech-tips/nvidia-tesla-m40-24gb-gpu-accelerator-maxwell-gm200-close
Complex shutdown - https://www.maketecheasier.com/schedule-ubuntu-shutdown
Sudo shutdown - https://sdet.ro/blog/shutdown-ubuntu-with-timer
Video trimmer - https://launchpad.net/~kdenlive/+archive/ubuntu/kdenlive-stable
Flatpak - https://flatpak.org/setup/Ubuntu

Installation commands

sudo snap install jupyter
sudo apt install git
sudo apt install nvidia-cuda-toolkit
sudo apt install curl
sudo apt install jq
sudo pip install liquidctl (to install fan controller globally)
sudo apt install net-tools
sudo apt install vim
sudo apt-get install htop
sudo apt install hddtemp
sudo apt install lm-sensors
pip install - user pipenv
sudo apt install pipenv
pipenv install jupyterlab

Installing Docker

sudo su (run from root)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update && sudo apt-get install -y - no-install-recommends docker-ce docker-ce-cli containerd.io
sudo apt-get install -y - no-install-recommends nvidia-docker2 nvidia-container-toolkit nvidia-container-runtime
sudo apt-get upgrade

Steps for Cuda upgrade
First removed existing:
sudo dpkg -P $(dpkg -l | grep nvidia-driver | awk '{print $2}')
sudo apt autoremove

then added new:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv - fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt update
sudo apt -y install cuda

then rebooted and do nvidia-smi

NVIDIA-SMI 510.47.03

Driver Version: 510.47.03

CUDA Version: 11.6

Thank you!

Beginner’s Guide To Machine Learning - Types of Machine Learning

Darren Broderick (DBro) — Thu, 02 Dec 2021 17:48:45 +0000

Source: chatbotslife

As a beginner in the world of Machine Learning (ML) I want to document the learnings I come across and explain them in a clear manner. Over a series of articles I plan to walkthrough different elements of Machine Learning from a beginner level up to intermediate.

This short article will look at only 2 things;

What is Machine learning?
Four Main Training processes (Supervised, Unsupervised, Semi-Supervised & Reinforcement).

What is Machine Learning (ML)?
ML is the process of solving a problem through the following steps;

Gathering a dataset (tabular data)
Building a statistical model based on the dataset
The inference(conclusion, or use of) that model to solve a practical problem.

A Small ML Example

Data gathered from a central server for traffic
A prediction model is built from each car dataset row, input data(location and speed)

To estimate upcoming congestion areas so that they can be prevented, a “model” is created.

A model is simply a file that identifies patterns(makes a prediction) based on the input data.

Machine learning in such scenarios helps to estimate the regions where congestion could be found.

Daily traffic experiences (data) is used to train a model to essentially see the future.

Machine Learning Types
Here are the four types of ML and a short description that helps us identify the differences later on.

Source: Simplilearn

Supervised
Using data that is labelled, when a dataset is in a collection that is labelled, each row of data is called a feature vector.

Unsupervised
Data here is unlabeled, during training the model discovers patterns and new information on its own.

Semi-supervised
The data here has only a small amount that is labelled with the rest being largely unlabelled, as a hybrid between the 2 above it is popular for classifying text documents.

Reinforcement
No data is given to the model is this type, training is very different for reinforcement as the process is random and the model makes sequences of decisions to try and get the highest return value possible. Very popular in gaming and a stable for AWS DeepRacer.

Supervised Learning
We mentioned “feature vector” for supervised but as an example, just imagine this data as the height, weight and gender of a person.
The most important part here is that each datarow has the same labels so they can be differentiated between their features.

An example of this would be spam detection for emails, you have 2 labels here; spam and not_spam.

Source: TowardsDataScience

So the goal of supervised learning is to use a dataset to produce a model that takes a feature vector as an input and outputs a prediction.
Unsupervised Learning
The difference to supervise learning is the data is unlabeled, and the goal is to find unknown patterns within your dataset.
An example of this would be for fraud detection, (or anomaly detection as the hierarchical term).

Source: Cloudera

It’s effective at detecting unseen events or rare occurrences by models made through unsupervised using non-anomalous training examples (datarows). i.e how different is event x from a “typical” example in the dataset.
It’s worth showing the differences between these 2 techniques before explaining semi-supervised.
Semi-Supervised Learning
So for semi the dataset contains both labeled and unlabeled examples. Usually unlabeled with have a much higher percentage over labeled. The outcome of semi is the same as supervised, that is, to use the labeled data to produce a model.
Then what’s the point of semi and having more unlabeled data?
You might think that extra unlabeled data will harm your model training but look at it this way; you’re actually adding more information about your problem.

Source: MLMinds

Given that the labeled data came from a similar sample-set as the unlabeled then in effect you’ve improved the probability distribution for your entire dataset to leverage.
If the data you are working with is challenging to label, consider semi-supervised learning to help ease the process.
Reinforcement Learning
Reinforcement learning (RL) is the training of machine learning models to make a sequence of decisions.
It is a subfield of ML where the machine or “agent” in this learning context learns in an environment that can ingest state as a vector of features.

Source: MathWorks

A policy is a function (similar to models in supervised learning)

Taking a feature vector of the state supplied by the environment, the agent produces an action to achieve maximum rewards (usually a float value). As the process is sequential, so to is the decision making. The long term goal of the RL lifecycle is to continually optimise the actions taking to achieve better rewards.

The agent learns to achieve a goal in an uncertain, potentially complex environment. In RL, an artificial intelligence faces a game-like situation.

The most popular gamified existence of RL is AWS DeepRacer.

Source: AWS

For DeepRacer its goal is to;

Navigate the virtual track (environment)
Gather info like car position, distance from centerline (state)
Try a random speed and direction (action)
Logs its result from the policy (reward)
Repeat process until termination count (episodes)

AWS Community Builders Program and Why You Should Apply

Darren Broderick (DBro) — Fri, 19 Mar 2021 16:12:48 +0000

Deadline to apply for this program is March 31st
The program accepts a limited number of members per year
Apply -> https://amazonmr.au1.qualtrics.com/jfe/form/SV_8wR79G4spteIvhX

This short article will explain;

-What the AWS Community Program is
-Why you should apply
-Tips for your application

I work for Liberty IT in Belfast, there are 4 people in my workplace who are also community builders and hopefully you can be as well!

What is the AWS Community Program?
"The AWS Community Builders program offers technical resources, mentorship, and networking opportunities to AWS enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the technical community."
It is a massive area of connections, information, tech talks, soft skill building, beta opportunities, support for innovations and certifications.
All of this is mostly contained in 2 areas, a private Slack Channel and a blog post site known as Dev.To, mine is https://dev.to/iamdbro
The length of duration in the program is 1 year, you can re-apply at the end of your term.
The program is free for all members.
Expectations
Program members are expected to join virtual calls, participate in mentorship opportunities, continue to share or produce educational and technical content, actively engage with and help build the AWS community, and demonstrate continued interest in learning more about AWS.

Why you should apply
For the many reasons listed below;
To increase all your AWS skills in general or better yet find the niche of your focus and get expert help from a SME (subject matter expert)
Connections - You've access to AWS tech leads in almost any tech or discipline, I've mainly made use of ML deployments and CDK.
Beta AWS services tests (I'm currently under NDA so can't disclose)
New writing experiences and a new audience of readers- dev.to
Great CDK tutorials
Opportunity to speak at Indy Meetup(Nov 2020)
Opportunity to speak at NYC ML - March 2021
AWS Cert Voucher - Take any AWS Cert exam for free
A free subscription to cloud academy
A Swag welcome kit including $500 AWS credits (see picture below)
Reinvent ticket at a discount
Asset pack for logos and banners for your social medias

Tips for your application

Talk about why you want to apply.

It's not about getting "the free stuff" but growing your personal skill set, expanding your career ambitions or taking a leap into a brand new area of tech you always wanted to explore.
Write about your current or completed projects to date, (helps to add your personal github or selective repos).
What future areas of tech you are looking to learn more about or greatly upskill in.

Mention any certifications you completed or are currently studying for.
Provide any example project ideas you have that you could get potential collaboration or feedback on.

Finally I just want to wish you good luck if you choose to apply and I hope to see you in the community.

Application Details - https://aws.amazon.com/developer/community/community-builders/

Building Modern Applications with AWS CDK (Session 1)

Darren Broderick (DBro) — Mon, 15 Feb 2021 22:08:02 +0000

This article is concise material from "AWS Dev Hour"
https://www.twitch.tv/videos/892197005 lasts just over an hour, this article will go through the main notes and has a separate working GitHub I'm building into.

https://github.com/DarrenBro/modernApp-AWS-CDK

There were however some issues I came across not covered by the video that I've included fixes for or tasks you should complete on your end.

Node Version 15+ had issues with deploying lambda assets https://github.com/aws/aws-cdk/issues/12536
Fix ->Downgrading node and deleting cdk.out/.cache should stop this issue from occurring.

https://stackoverflow.com/questions/8191459/how-do-i-update-node-js
If you continue without completing steps not shown in the video for getting pillow(PIL) into your directory (steps shown below title Pillow) you'll get the following CloudWatch log error;
Fix -> I've included the necessary extracted zip file and put it in the project under the folder "reklayer".

The video misses out showing the creation of the lambda layer. This allows the lambda to import 'PIL' from your attached zip file you created.

Fix -> I've attached code snippets that need included in cdkMainStacks.ts.

Otherwise you'll get the below error in your CloudWatch logs.

'index' is referring to the main body of the rekognition lambdaI've flooded cdkMainStacks.ts with comments better explaining different components. This file is the main area that most of the code will get added.

For me this has been a great way to get more hands on with CDK with good progression intervals and building it out yourself is the best way to get familiar with any tech, plus finding solutions for these errors has helped commit it more to memory.

Tech Stack

aws-cli
https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html

Node.js to run the project (Version 10.3.0–14.15.3)
https://nodejs.org/en/download/

IDE that can run any of these languages;
Typescript, Javascript, Java, C# or Python
(I'm using IntelliJ with Typescript)

AWS CDK toolkit and bootstrap

Details of the App

Built using CDK

User will be able to upload a photo through a react JS UI, stored in s3 (upcoming sessions).

It will then trigger a lambda upon the s3 action and store it in dynamoDB (which stores metadata and labels, done this session)

We can then query(scan) the db to check labels for the image have been added after going through rekognition service.

All graphics are taken from the AWS run through video.

Session 1 will consist of;
Installing prerequisites
Explaining what AWS CDK is
Creating a new project and getting dependencies installed
Building 3 components -> s3 / lambda & dynamo db
Copying an image to scan and scanning the table

What is AWS CDK?
A framework we can use to declare our AWS resources
Sits on top of Cloud Formation
(CDK produces the CFT Template after synthesise "cdk synth")
Simplifies the process of building out these resource templates

This is what an application looks like in the CDK, starting with the App, and broken into stacks and constructs(which is what will be built later on).
Steps to get started;
"mkdir cdk-project"
"cdk init" (shows templates available)
"cdk init app -l=typescript" (Go with typescript)
Install aws-iam "npm i @aws-cdk/aws-iam"

Notes
app initialised in cdk-projects.ts
cdk.json tells cdk toolkit how to run the project

Pillow
The AWS Lambda function uses the Pillow library for the generation of thumbnail images. This library needs to be added into our project so that we can allow the CDK to package it and create an AWS Lambda Layer for us. To do this, you can use the following steps.
Launch an Amazon EC2 Instance (t2-micro) using the Amazon Linux 2 AMI
SSH into your instance and run the following commands:

sudo yum install -y python3-pip python3 python3-setuptools
python3 -m venv my_app/env
source ~/my_app/env/bin/activate
cd my_app/env
pip3 install pillow
cd /home/ec2-user/my_app/env/lib/python3.7/site-packages
mkdir python && cp -R ./PIL ./python && cp -R ./Pillow-8.1.0.dist-info ./python && cp -R ./Pillow.libs ./python && zip -r pillow.zip ./python
Copy the resulting archive 'pillow.zip' to your development environment (we used an Amazon S3 bucket for this)
Extract the archive into the 'reklayer' folder in your project directory

Your project structure should look something like this:
project-root/reklayer/python/PIL
project-root/reklayer/python/Pillow-8.1.0.dist-info
project-root/reklayer/python/Pillow.libs
Remove the python.zip file to clean up
Terminate the Amazon EC2 Instance that you created to build the archive

1st component to build is s3
npm i @aws-cdk/aws-s3
In cdkMainStack.ts we'll be adding;
import s3 = require('@aws-cdk/aws-s3');
const imageBucket = new s3.Bucket(this, imageBucketName, {
removalPolicy: cdk.RemovalPolicy.DESTROY
})
new cdk.CfnOutput(this, 'imageBucket', {value: imageBucket.bucketName});
A quick note on the constructs here.
So for s3 code above it's the same principle.

2nd component to build is dynamoDB for storing the image labels
npm i @aws-cdk/aws-dynamodb
const imageTable = new dynamodb.Table(this, 'ImageLabels', {
partitionKey: {name: 'image', type: dynamodb.AttributeType.STRING},
removalPolicy: cdk.RemovalPolicy.DESTROY
});
new cdk.CfnOutput(this, 'cdkTable', {value: imageTable.tableName});

3rd component to build is Lambda layer and Lambda
Layer needed to import the PIL library when rekognition lambda is executed.
const layer = new lambda.LayerVersion(this, 'pil', {
code: lambda.Code.fromAsset('reklayer'),
compatibleRuntimes: [lambda.Runtime.PYTHON_3_7],
license: 'Apache-2.0',
description: 'A layer to enable the PIL library in our Rekognition Lambda',
});
Lambda's job is to pull an image from the s3 bucket, take the image and send it to the rekognition service to do the image detection.
npm i @aws-cdk/aws-lambda @aws-cdk/aws-lambda-event-sources
const rekognitionLambdaFunc = new lambda.Function(this, 'rekognitionFunction', {
code: lambda.Code.fromAsset('rekognitionlambda'),
runtime: lambda.Runtime.PYTHON_3_7,
handler: 'index.handler',
timeout: Duration.seconds(30),
memorySize: 1024,
layers: [layer],
environment: {
"TABLE": imageTable.tableName,
"BUCKET": imageBucket.bucketName,
"RESIZEDBUCKET": resizedBucket.bucketName
}
});

Last part to add is Permissions and Roles
To trigger Lambda when object(image) is created in S3.
rekognitionLambdaFunc.addEventSource(new event_sources.S3EventSource(imageBucket, {events: [s3.EventType.OBJECT_CREATED]}))
Permission to read from s3.
imageBucket.grantRead(rekognitionLambdaFunc);
resizedBucket.grantPut(rekognitionLambdaFunc);
Permission to allow the result of rekognition service from the sent image to be stored in dynamoDB.
imageTable.grantWriteData(rekognitionLambdaFunc);
Permission policy to allow label detection from rekognition across all resources.
rekognitionLambdaFunc.addToRolePolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
// permission policy to allow label detection from rekognition across all resources
actions: ['rekognition:DetectLabels'],
resources: ['*']
}))

And make sure you've updated your bucket name to the same as the stack. (Would recommend changing prefix to your name, but stack with add a unique suffix ID value in template after synthesising).

To create your template for your stacks run:
cdk synth
A set of files in a folder called "cdk.out" will be produced with assets and your json template file.
You can re-run cdk synth to populate any code updates.
synth will manufacture a stack (our components) down into fully formed json yaml template (CloudFormation template) which will include all the resources we added above.
When you see the generated template it will already be quite large. Already cdk has saved us a bunch of effort!

To deploy the resources that we just created run the below command.

cdk deploy
After seeing the below you just want to hit 'y'
This should take around a few minutes to complete.
Result of "cdk deploy" -> ✅ cdkMainStack

Output example
cdkMainStack.cdkTable = cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP

cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh

You can also view your CF stacks in AWS console straight away, pretty nice!

So that's your s3, lambda and dynamo db all created for you.
A quick note on the lambda. For the clients below:
As this lambda is being invoked we don't have to re-instantiate these above clients every time the lambda is run, it can stay 'hot' and this helps with performance.

Test the image upload (now that s3 resources have been created using cdk deploy)

You'll need to use your unique bucket name, e.g.
aws s3 cp testimage.jpg s3://cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh

Logs

Check CloudWatch logs for events and for any errors
In log group "/aws/lambda/cdkMainStack……" -> click latest log stream to see the image being processed.

Or you can check your new dynamoDB table in the console.

These labels are from an image of Stargate's Atlantis' wormhole after it was sent through AWS Rekognition
Or you can scan the dynamoDB table, using the output name from the cdk synth. e.g.

aws dynamodb scan --table-name cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP

And you should see a single output of your image in the terminal

To get rid of all resources and stop any future running costs run:
cdk destroy
(which is also why we added to dynamoDB)
removalPolicy: cdk.RemovalPolicy.DESTROY

However, even with this policy added to S3 and using cdk destroy the formation will have a status "DELETE_FAILED"
Without the policy added you will see that the CloudFormation stack is deleted successfully but the S3 bucket remains. Why?

By default, the Construct that comes from the S3 package, has a default prop called removalPolicy: cdk.RemovalPolicy.RETAIN.
(CloudFormation does not destroy buckets that are not empty).

Option 1: Manually clean the bucket contents before destroying the stack

You can do this from the AWS S3 user interface or through the command line, using the AWS CLI:

Cleanup bucket contents without removing the bucket itself;

aws s3 rm s3://bucket-name --recursive

Then run;

cdk destroy

Then the cdk destroy will proceed without errors. However, this can quickly become a tedious activity if your stacks contain multiple S3 buckets or you use stacks as a temporary resource so some automation would help (option 2).
Option 2: Automatically clear bucket contents and delete the bucket

A 3rd party package called @mobileposse/auto-delete-bucket provides a custom CDK construct that wraps around the standard S3 construct and internally uses the CloudFormation Custom Resources framework, to trigger an automated bucket contents cleanup when a stack destroy is triggered.

Install the package:
npm i @mobileposse/auto-delete-bucket

Use the new CDK construct instead of the standard one:
import { AutoDeleteBucket } from '@mobileposse/auto-delete-bucket'
const bucket = new AutoDeleteBucket(this, 'my-data-bucket')

Summary

Add components / build / deploy

cdk synth
cdk deploy
(If you makes any changes, you can run another 'cdk deploy')

Take note of outputs (example)

cdkMainStack.cdkTable = cdkMainStack-ImageLabels
cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucket

Copy test image

aws s3 cp testimage.jpg s3://{bucketName}

Check table has been populated, if not check logs

aws dynamodb scan --table-name {tableName}

Cleanup tasks

aws s3 rm s3://{bucketName} --recursive
cdk destroy

That's all for this session, take care!

AWS DeepRacer

Darren Broderick (DBro) — Wed, 28 Oct 2020 13:46:59 +0000

The DeepRacer car or ‘agent’ as it’s also referred to is a fully autonomous race car, programmed by us in python and trained across many iterations in AWS SageMaker on a simulation environment spun up by AWS RoboMaker.
We don’t provide the training data upfront like in supervised and unsupervised machine learning, neither do we apply any labels initially.
Instead the agent supplies its own timed delay label, known as the ‘reward’.

The data is gathered by the agent’s photo lens which are turned to greyscale. These images are off the simulated track. It tries an ‘action’ (JSON format with properties of speed and angle), you set these before training. Then analyses rewards received for its attempts, and repeats the process with different actions to look for greater rewards, being a returned as a float number in your reward function (will discuss later).
In short, the agent’s single focus is; return the maximum rewards possible.

Race League and Rules

The rules are simple. Everyone gets 4 minutes to achieve their best lap time on the re:Invent track. You’re allowed to come off the track a maximum of 3 times in order to qualify a lap. But each “off course” must be fixed by manually re-plotting the car back on the track, eating into your lap time.

Reinforcement Learning

Reinforcement Learning(RL) is a type of machine learning technique that enables an agent to learn in an interactive environment by trial and error using feedback from its own actions and experiences.
Unlike supervised learning where feedback provided to the agent is a correct set of actions for performing a task, reinforcement learning uses rewards and punishment as signals for positive and negative behaviour.

Compared to unsupervised learning, reinforcement learning is different in terms of goals. While the goal in unsupervised learning is to find similarities and differences between data points, in reinforcement learning the goal is to find a suitable action model that would maximise the total cumulative reward of the agent. The figure to the left represents the basic idea and elements involved in a reinforcement learning model.

Environment: Physical world in which the agent operates.
State: Current situation of the agent.
Reward: Feedback from the environment.
Policy: Method to map agent’s state to actions.
Value: Future reward that an agent would receive by taking an action in a particular state.
SageMaker: With each batch of experiences from RoboMaker, SageMaker updates the neural network, “and hopefully your model has improved.”

Tips and Tricks

Keep your models simple, the model above focuses on keeping the car on the centre line, it is a great place to start with, wouldn’t change a thing above as a beginner.

Don’t reward with a negative float, it can force the car to finish laps early to avoid them and cuts valuable training episodes. Punish instead with multiplying by decimal values.

Get a look at the logs of your training. They are also on cloudWatch but not very readable.
https://codelikeamother.uk/using-jupyter-notebook-for-analysing-deepracer-s-logs

Train your models for 1–2 hours, you can clone them to continue further training, but 1-2 hours gives a good indication if you are making progress on the track. (See on left).

My Experience

I’ve really enjoyed the DeepRacer experience as a fun competition but more of a way into understanding RL and machine learning in general. It’s taken a lot of time to get through the vast material but well worth the learning journey, if you are interested in learning more just let me know!
The best way to get involved is to race a model you’ve made yourself, then you’re hooked, which is a good thing.
I plan to write an enhanced deep racer guide in the future to focus on ways to be competitive and efficient with your time, hopefully they work for us at the end of October :)
Thank you!

Useful Resources

Getting Started
https://aws.amazon.com/deepracer/getting-started/

AWS Guide
https://github.com/aws-samples/aws-deepracer-workshops/blob/master/Workshops/2019-AWSSummits-AWSDeepRacerService/Lab1/Readme.md

Build your own track https://medium.com/@autonomousracecarclub/guide-to-creating-a-full-re-invent-2018-deepracer-track-in-7-steps-979aff28a6f5

AWS Training
https://www.aws.training/Details/eLearning?id=32143