DEV Community: vindianadoan

Fixing AWS CLI Version Conflicts on Windows

vindianadoan — Wed, 12 Mar 2025 02:05:06 +0000

Symptom: You install AWS CLI v2 but aws --version still shows v1.x.x.

Cause: An older aws.exe appears earlier in your PATH than the new CLI. On Windows, the system uses the first match it finds.

Quick Check

Run:

where aws

You might see:

C:\Users\YourName\anaconda3\Scripts\aws
C:\Users\YourName\anaconda3\Scripts\aws.cmd
C:\Program Files\Amazon\AWSCLIV2\aws.exe

The first path is always used (in this example, Anaconda’s AWS CLI v1).

Two Fixes

1. Remove the Old CLI

If installed via conda:

conda remove awscli

If installed via pip:

pip uninstall awscli

Then, confirm by running:

where aws

You should now see only:

C:\Program Files\Amazon\AWSCLIV2\aws.exe

And aws --version should show 2.x.x.

2. Reorder PATH

If you can’t remove the old CLI:

Open System Properties → Environment Variables → Path.
Move: C:\Program Files\Amazon\AWSCLIV2\ above any Python or Anaconda paths.
Restart your terminal and run aws --version again.

Setting anaconda Python as the default

vindianadoan — Wed, 12 Mar 2025 01:42:52 +0000

Just documenting a quick note for me to refer back later.

Problem

When I was trying to check the AWS CLI version on my terminal, I get this warning in return:

which made me go: "Hang on, pretty sure I have anaconda installed".

It must be because when I was installing anaconda, I didn't pick the setting so that Python is added to the environment.

How to Fix It

Method 1: Run Command Prompt as Administrator:

Type the following commands, substituting the path to your Anaconda Python:

assoc .py=Python.File
ftype Python.File="C:\Users\YourName\anaconda3\python.exe" "%1" %*

These commands tell Windows to associate .py with a “Python.File” type, and then map “Python.File” to your Anaconda Python executable.

Method 2: Use the Windows Default Apps Settings (No Admin Required):

Open Settings → Apps → Default apps.
Click Choose default apps by file type and scroll to .py.
Select the current default app (or “No default”) and browse to your Python interpreter (e.g. C:\Users\YourName\anaconda3\python.exe).

Now it's gone.

[Boost]

vindianadoan — Mon, 10 Mar 2025 06:53:54 +0000

Part 1: Deploying an R Model as a Containerised API with Docker, Plumber, and renv

vindianadoan ・ Mar 10

Part 1: Deploying an R Model as a Containerised API with Docker, Plumber, and renv

vindianadoan — Mon, 10 Mar 2025 02:54:51 +0000

In this post, I’ll walk you through the entire process of building an R model, containerising it as a RESTful API using the Plumber package, and adding detailed logging. We’ll cover every step, including troubleshooting common issues (like missing system libraries) and ensuring reproducibility with renv. By the end of this tutorial, you’ll have a Docker image that serves your model via an API endpoint—ready for local testing or deployment to cloud services like AWS. In this part, we will build the components required for local testing, deployment to AWS will come in part 2.

You can find the entire code and Dockerfiles in the GitHub repo.

Overview & Prerequisites
Building & Saving an R Model
Building the Base Image with System Dependencies and renv
Iterative Model Image for Rapid Development
Automating Docker Builds with a Shell Script
Managing R Dependencies with renv
Creating a Plumber API Endpoint
Testing the API
- Running a Container of the Built Image
- Using Curl
- Using Postman
- Using R’s httr Package
Troubleshooting
- Package Installation Issues
- Docker Volume Mount
- Interact with Running Container

1. Overview & Prerequisites

In this guide, we build a predictive model using the classic iris dataset and train a random forest classifier. We then containerise our R code with Docker and expose a RESTful API via Plumber. We’ll add logging using the logger package and manage dependencies with renv for reproducibility.
Key components include:

R Model Training: Using randomForest to train on the iris dataset.
Containerisation: Creating a reproducible R environment using Docker.
Dependency Management: Using renv to lock package versions.
Plumber API: Exposing a /predict endpoint with Swagger documentation.
Logging: Detailed logging of API requests and responses, with logs mounted on the host.

To effectively monitor our deployment workflow, I have established a dual logging system. The outside_container_logs directory captures logs generated during local development—this includes running and testing individual R scripts, building the Docker container, and future deployments. Meanwhile, logs from processes running inside the container are directed to inside_container_logs, which is mounted to /app/logs within the container. This separation enables us to clearly differentiate between pre-deployment activities and runtime behavior inside the container, allowing us to efficiently troubleshoot and ensure that both our development and production environments are operating as expected.

Before you begin, ensure you have the following installed:

Docker: For container building.
R (4.4.1 or compatible): For model training and package management.
renv: For dependency management.
Plumber and logger packages: To expose the API and manage logs.

2. Building & Saving an R Model

First, we create a training script (train.R) that trains a random forest model on the iris dataset and logs each step. This script automatically detects whether it’s running inside a container (by checking for /app/logs) and sets up the appropriate logging directory.
train.R

# train.R
library(randomForest)
library(logger)

# Choose the log directory based on whether /app/logs exists (inside container) or not (outside)
log_dir <- if (dir.exists("/app/logs")) {
  "/app/logs/R_logs"
} else {
  "logs/outside_container_logs/R_logs"
}

# Ensure the directory exists; create it if necessary
if (!dir.exists(log_dir)) {
  dir.create(log_dir, recursive = TRUE)
}

# Create a timestamp for the log file name
timestamp <- format(Sys.time(), "%Y%m%d_%H%M%S")
log_file <- file.path(log_dir, paste0(timestamp, "_training.log"))

# Configure logger with a unique file name for each run
log_appender(appender_file(log_file))

log_info("Starting model training...")

# Train a random forest model to classify iris species
data(iris)
model <- randomForest(Species ~ ., data = iris, ntree = 100)

log_info("Model training complete.")

# Capture the output of print(model)
model_output <- capture.output(print(model))
# Log the captured output
log_info("Model details:\n{paste(model_output, collapse = '\n')}")

# Save the trained model to disk
saveRDS(model, file = "data/model.rds")
log_info("Model saved to model.rds.")

Run this script (manually or as part of your Docker build) to generate your model and record the training process.

3. Creating a Plumber API Endpoint

Next, create an API using Plumber. We’ll define an endpoint /predict that:

Loads the pre-trained model.
Parses the incoming JSON request.
Makes a prediction.
Logs request and response details.

serve.R

# serve.R
library(plumber)
library(randomForest)
library(logger)
library(jsonlite)

# Set up logging
log_dir <- if (dir.exists("/app/logs")) {
  "/app/logs/R_logs"
} else {
  "logs/outside_container_logs/R_logs"
}
if (!dir.exists(log_dir)) {
  dir.create(log_dir, recursive = TRUE)
}
# Create a timestamp for the log file name
timestamp <- format(Sys.time(), "%Y%m%d_%H%M%S")
log_file <- file.path(log_dir, paste0(timestamp, "_request.log"))
log_appender(appender_file(log_file))

# Load the trained model
model <- readRDS("data/model.rds")

#* @filter log_requests
function(req, res) {
  log_info("Incoming request: {req$REQUEST_METHOD} {req$PATH_INFO}")
  log_info("Request body: {req$postBody}")
  plumber::forward()
}

#* @post /predict
#* @serializer json
function(req) {
  data <- fromJSON(req$postBody)
  input_data <- as.data.frame(data)
  pred <- predict(model, input_data)
  log_info("Prediction: {paste(pred, collapse=', ')}")
  list(prediction = as.character(pred))
}

Plumber automatically generates Swagger documentation (accessible at http://localhost:8080/__docs__/) from these annotations.

4. Building the Base Image with System Dependencies and renv

To ensure that the heavy lifting of installing system dependencies and R packages is done only once, we created a base image. This image includes R 4.4.1, necessary system libraries (for packages such as textshaping and ragg), and restores R packages using an renv.lock file.
Dockerfile.base

# Use a specific R base image
FROM r-base:4.4.1

# Install system dependencies required by R packages
RUN apt-get update && apt-get install -y \
    libcurl4-openssl-dev \
    libssl-dev \
    libxml2 \
    libxml2-dev \
    libz-dev \
    libsodium-dev \
    libfontconfig1-dev \
    libfreetype6-dev \
    libharfbuzz-dev \
    libfribidi-dev \
    libpng-dev \
    libtiff5-dev \
    libjpeg-dev \
    build-essential \
    pkg-config \
    wget \
    apt-transport-https \
    ca-certificates \
 && rm -rf /var/lib/apt/lists/*

# Create a directory for your application
WORKDIR /app

# Copy your renv lock file to restore R packages
COPY renv.lock /app/renv.lock

# Install renv and restore packages using renv.lock
RUN R -e "install.packages('renv', repos='http://cran.rstudio.com/'); \
          renv::restore(repos = c(CRAN = 'http://cran.rstudio.com/'))"

This image (tagged as, for example, r-base-mlops:4.4.1) serves as a stable foundation.

5. Iterative Model Image for Rapid Development

Building on the base image, we created a smaller model iteration image that contains only our model code. This image allows us to rapidly iterate on our model (or API) without reinstalling all the system dependencies and R packages every time.
We’ll create a Docker image that:

Uses the R base image (here, version 4.4.1).
Copies your scripts and model into the container.
Exposes the Plumber API on port 8080.

Dockerfile.model

# Use the prebuilt base image that contains R 4.4.1 and all system dependencies & renv packages
FROM r-base-mlops:4.4.1

# Set the working directory
WORKDIR /app

# Create directories for logs and model data
RUN mkdir -p /app/logs/docker_logs /app/logs/R_logs /app/logs/aws_logs
RUN mkdir -p data

# Copy your model scripts into the container
COPY ./R/train.R /app/train.R
COPY ./R/serve.R /app/serve.R
# Copy the runner script to start the API
COPY ./R/run_api.R /app/run_api.R

# Run the training script to generate or update your model (e.g. data/model.rds)
RUN Rscript train.R

# Expose the port on which your Plumber API will run
EXPOSE 8080

# Start the Plumber API using the runner script
CMD ["Rscript", "run_api.R"]

The run_api.R script simply loads serve.R and starts the API:

# run_api.R
pr <- plumber::plumb("serve.R")
pr$run(host = "0.0.0.0", port = 8080)

This setup ensures that when you change your model code (in train.R or serve.R), you only need to rebuild the model image—not the entire base image.

6. Automating Docker Builds with a Shell Script

To build the Docker images, you can run these command lines:

# 1. Build the Docker base image 
docker build --no-cache -f docker/Dockerfile.base -t r-base-mlops:4.4.1 .
# 2. Build the Docker model image 
docker build --no-cache -f docker/Dockerfile.model -t my-r-sagemaker-model .

However, to simplify the process of building your Docker images above and ensure thorough logging, I’ve created an automation script. This script lets you choose between building the model image or the base image with a simple command-line argument. It also saves detailed build logs for later review.
docker_build.sh

#!/bin/bash

# Change working directory to the repository root.
# This works if your script is in ./bash/ and your repo root is one level up.
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$REPO_ROOT" || exit 1

# Check if an argument is provided
if [ $# -eq 0 ]; then
  echo "Usage: $0 <model|base>"
  exit 1
fi

container_type=$1

# Ensure the logs directory exists
mkdir -p logs/outside_container_logs/docker_logs/

# Create a unique log filename using a timestamp and the container type
timestamp=$(date +'%Y%m%d_%H%M%S')
log_file="logs/outside_container_logs/docker_logs/${timestamp}_${container_type}_build.log"

echo "Building starting. Build log will be saved to $log_file"

# Check the argument to determine which Docker build command to use
if [ "$container_type" = "model" ]; then
    docker build --no-cache -f docker/Dockerfile.model -t my-r-sagemaker-model . > "$log_file" 2>&1
elif [ "$container_type" = "base" ]; then
    docker build --no-cache -f docker/Dockerfile.base -t r-base-mlops:4.4.1 . > "$log_file" 2>&1
else
    echo "Unknown container type: $container_type. Use 'model' or 'base'."
    exit 1
fi

echo "Build log saved to $log_file"

To run this script:

# To give it permission
chmod +x ./bash/docker_build.sh 
# To build the base image 
./bash/docker_build.sh base
# To build the model image 
./bash/docker_build.sh model

7. Managing R Dependencies with renv

Using renv ensures that your container installs the exact versions of R packages required by your project. To set this up locally:

# 1. Initialise renv:
renv::init()
# 2. Install any packages you use (eg. plumber, randomForest, logger, tidyverse). 
# 3. Snapshot the environment:
renv::snapshot()

8. Testing the API

Running a Container of the Built Image

Normally, to run a container, this is all you need to run:

docker run <your_container_image>

Let's say our container image name is called my-r-sagemaker-model.

But since we want to expose the container's API specifically to port 8080, we need to explicitly map the container's port 8080 to port 8080 on our host machine.

docker run -p 8080:8080 my-r-sagemaker-model

Without the -p 8080:8080 flag, the container will still run, but Docker does not automatically expose any ports to your host. This means that even though your API is running inside the container on port 8080, you wouldn’t be able to connect to it from your host machine.

Finally, since we also want to mount the app/logs/ folder within the container with our external log folder (/logs/inside_container_logs):

docker run -p 8080:8080 -v "$(pwd)/logs/inside_container_logs:/app/logs" my-r-sagemaker-model

Testing the Container Using Curl

curl -X POST "http://localhost:8080/predict" \
     -H "Content-Type: application/json" \
     -d '{"Sepal.Length": 5.1, "Sepal.Width": 3.5, "Petal.Length": 1.4, "Petal.Width": 0.2}'

You should be getting this result:

Prediction: setosa

Testing the Container Using Postman

Set the HTTP method to POST.
Enter the URL: http://localhost:8080/predict.
Under the Headers tab, add Content-Type for Key with value application/json.
In the Body tab, select raw and choose JSON from the dropdown.
Paste the following JSON:

   {
     "Sepal.Length": 5.1,
     "Sepal.Width": 3.5,
     "Petal.Length": 1.4,
     "Petal.Width": 0.2
   }

Click Send to view the response.

Testing the Container Using R’s httr Package

library(httr)
library(jsonlite)

url <- "http://localhost:8080/predict"
input <- list(
  Sepal.Length = 5.1,
  Sepal.Width  = 3.5,
  Petal.Length = 1.4,
  Petal.Width  = 0.2
)
response <- POST(
  url,
  body = toJSON(input, auto_unbox = TRUE),
  encode = "raw",
  add_headers("Content-Type" = "application/json")
)
result <- content(response, as = "parsed")
print(result)

9. Troubleshooting

Container fails to install a package

If a package fails to install (e.g., textshaping or ragg), check the error message for missing system libraries. Update your Dockerfile.base to install the required development packages (such as libharfbuzz-dev, libfribidi-dev, libpng-dev, libtiff5-dev, and libjpeg-dev).
If you don't have the logs setup, you can check by running the container, then call the library to see if it's installed, and try to reinstall it inside the container to see if it's having any issues:

Run the container interactively:

docker run -it my-r-sagemaker-model R

Then you will be in the R console. Here you can check if something is installed (Eg. plumber):

library(plumber)

If the package wasn't installed correctly, it will say there's no package. Here you can try to reinstall it to see what error comes out, if it's an issue with missing system requirements, it should tell you what is missing, then you just need to that package's installation to the Dockerfile.base script. If the package loads without error, the issue is resolved. To exit, type:

q()

and then press n when prompted to save the workspace.

Docker Volume Mount

When running your container, use the Docker -v (or --volume) flag to map the host directory to the container directory. For example, if your host directory is in the current working directory, you could run:

docker run -v "$(pwd)/logs/inside_container_logs:/app/logs" your_container_image

This command tells Docker to mount your host’s logs/inside_container_logs directory into the container at /app/logs.
After running the container, use docker container inspect <container-id> to verify that the mount is correctly set, and check the local folder to ensure that logs are being written there.
Working on Windows brought its own challenges. When mounting a host directory to capture logs, my Docker run command was appending extra characters like ;C to the path. For example, instead of a clean path, I saw:

"Mounts": [
            {
                "Type": "bind",
                "Source": "C:\\Users\\Vince\\repos\\AWS-projects\\deploy-R-model-to-AWS-ECR\\logs\\inside_container_logs;C",
                "Destination": "\\Program Files\\Git\\app\\logs",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }

This was due to Git Bash’s interpretation of Windows paths, so it's recommended to use another terminal/powershell for this.

Any logs written to /app/logs by your R scripts (e.g., logs/R_logs/training.log and logs/R_logs/requests.log) will now be saved in your local logs folder (logs/inside_container_logs/R_logs), allowing you to access them even after the container stops.

Interact with Running Container

If you want to interact directly with a running container, for instance, you need to inspect logs directly within a running container without mounting a volume, you can use the docker exec command:

docker exec -it <container-id> bash
ls /app/logs

For example, I first call docker ps to get the container-id, then run the above docker exec command to enter the container, where you can do various things with it, such as inspecting the logs:

For example:

Vince@tanjiro [~/.../AWS-projects/deploy-R-model-to-AWS-ECR] ± master ✓           [12:35:08]> docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED       STATUS
     PORTS                    NAMES
04221d808cc1   my-r-sagemaker-model:latest   "R -e 'pr <- plumber…"   4 hours ago   Up 4 hours   0.0.0.0:8080->8080/tcp   distracted_nobel
Vince@tanjiro [~/.../AWS-projects/deploy-R-model-to-AWS-ECR] ± master ?:1 ✗       [16:31:35]1 > docker exec -it 04221d808cc1 bash
root@04221d808cc1:/app# ls
data  logs  renv.lock  serve.R  train.R
root@04221d808cc1:/app# cd logs/
root@04221d808cc1:/app/logs# ls
R_logs
root@04221d808cc1:/app/logs# cd R_logs/
root@04221d808cc1:/app/logs/R_logs# ls
20250310_012557_request.log
root@04221d808cc1:/app/logs/R_logs#
exit

What's next:
    Try Docker Debug for seamless, persistent debugging tools in any container or image → docker debug 04221d808cc1
    Learn more at https://docs.docker.com/go/debug-cli/

A cheat sheet on Pandas DataFrame Attributes

vindianadoan — Sat, 18 Jan 2025 16:04:54 +0000

This post summarises a range of commonly used attributes of a pandas DataFrame object.
Using this DataFrame as an example:

import pandas as pd 
df = pd.DataFrame({
    'name': ["A", "B"],
    'value': [1,2]
})

1. Core Attributes

Attribute	Description	Output
`.shape`	Returns a tuple representing the dimensionality of the DataFrame in the form `(number_of_rows, number_of_columns)`.	`print(df.shape)` `# Output: (2,2)`
`.columns`	Returns an `Index` object containing the column labels of the DataFrame	`print(df.columns)` `# Output: Index(['name', 'value'], dtype='object')`
`.index`	Represents the row labels of the DataFrame, encapsulated in an `Index` object.	`print(df.index)` `# Output: RangeIndex(start=0, stop=2, step=1)`
`.dtypes`	Provides the data type of each column in the DataFrame.	`print(df.dtypes)` `# Output` `name object` `value int64` `dtype: object`

2. Data Inspection Attributes

Attribute	Description	Output
`.values`	Returns a NumPy array representation of the DataFrame's data.	`print(df.values)` `# Output:` `[['A' 1]` `['B' 2]]`
`.size`	Returns the total number of elements in the DataFrame (calculated as rows x columns).	`print(df.size)` `# Output: 4`
`.ndim`	Returns the number of dimensions of the DataFrame. For DataFrames, this is always 2.	`print(df.ndim)` `# Output: 2`
`.empty`	Returns `True` if the DataFrame is empty (ie. has no elements), otherwise `False`.	`print(pd.DataFrame().empty)` `# Output: True`

3. Others

Attribute	Description	Output
`.T` (Transpose)	Returns the transpose of the DataFrame, swapping rows with columns.	`print(df.values)` `name A B` `value 1 2`

ARIMA Modelling: Theory

vindianadoan — Wed, 24 May 2023 19:13:12 +0000

Introduction

ARIMA models, standing for AutoRegressive Integrated Moving Average, is a great model for predicting temporal time series data with a complex nature. Unlike basic regression models, which may struggle to capture complexity with just a few variables (usually due to autocorrelation, trend and seasonality), ARIMA excels in treating autocorrelation, trend and seasonality through a built-in differencing function (the "Integrated" part of ARIMA); thus, it removes the need for specifying explanatory variables and extra transformation. Examples of application include finance, weather predictions, and even anticipating website traffic.

Define the model

Let's build the ARIMA model from scratch. As discussed, the ARIMA model is made of three parts: an autoregressive part, an integrated part and a moving average part.

For example, given some time series

X = {X_1, X_2, ...}

The autoRegressive (AR) component says that an observation

X_t)

at certain point in time

t

, can be described as a linear combination of its lagged observations (ie. prior time points):

Xt=α1Xt−1+α2Xt−2...X_t = \alpha_1 X_{t-1} + \alpha_2 X_{t-2} ...

where

αi\alpha_i

are the parameters or coefficients of such regression.
(Note: if you find the terms linear combination and linear regression confusing, a linear regression is model that outputs predictions based on a linear combination of input features).
Using the lag operator (

L^pX_t = X_{t-q}

), you can also express the above as:

X_t = \alpha_1 (X_{t-1}) + \alpha_2 (X_{t-2}) ... + \alpha_p (X_{t-p})

X_t = \alpha_1 LX_{t} + \alpha_2 L^2X_{t} ... + \alpha_p L^pX_{t}

X_t = (\sum_{i=1}^{p} \alpha_i L^i)X_{t} \hspace{1cm} (1)

An optimisation of the AR part for the ARIMA model is an optimisation of the number of previous terms - or the number of time lags to be included in the linear combination. We call

p

the order of the autoregressive model.
For example, when we say the AR part has an order of two, we symbolise it as

A R (2)

, and express it as:

Xt=α1LXt+α2L2XtX_t = \alpha_1 LX_{t} + \alpha_2 L^2X_{t}

Now, equivalently, $X_t)$ can also be expressed as a combination of previous error terms. This is described in the Moving Average part of the ARIMA model:

X_t = \epsilon_t + \theta_1\epsilon_{t-1} + \theta_2\epsilon_{t-2} + ... + \theta_q \epsilon_{t-q}

X_t = \epsilon_t + \theta_1 L \epsilon_{t} + \theta_2 L^2\epsilon_{t} + ... + \theta_q L^q \epsilon_{t}

X_t = \epsilon_t + (\sum_{i=1}^{q} \theta_q L^q)\epsilon_{t} \hspace{1cm} (2)

where:

θi\theta_i

are the parameters/ coefficients of the linear combination of the error terms

ϵt\epsilon_t

are the error terms at time

t

q

is the order of this moving average.
Note that the concept of moving average here can be misleading, as moving average in this context simply refers to the moving window of the previous errors, it does not take an average of anything.

Since both (1) and (2) describe $X_t$ , we can add them together:

X_t = (\sum_{i=1}^{p} \alpha_i L^i)X_{t} + \epsilon_t + (\sum_{i=1}^{q} \theta_q L^q)\epsilon_{t}

This is often expressed as:

X_t - (\sum_{i=1}^{p} \alpha_i L^i)X_{t} = \epsilon_t + (\sum_{i=1}^{q} \theta_q L^q)\epsilon_{t}

\sum_{i=1}^{p} \alpha_i L^i)X_{t} = (1 + \sum_{i=1}^{q} \theta_q L^q)\epsilon_{t}

We say that, given time series data $X_t$ where $t$ is an integer index and the $X_t$ are real numbers, an $A R I M A (p, q)$ model is given by:

\sum_{i=1}^{p} \alpha_i L^i)X_{t} = (1 + \sum_{i=1}^{q} \theta_q L^q)\epsilon_{t}

Why do we need both an AR component and an MA component?

Since the AR(p) component is a combination of the past 'p' values, and the values of the series become more and more dependent on its past values when moving along the time series. Hence, it captures the trend and autocorrelation in the series.
Meanwhile, the MA(q) part is a combination of the past 'q' forecast errors, so it captures the "shock" (unexpected changes) to the model instead. The impact of a shock in the series decreases over time (called "shock decay"). Thus, the effect of an error made at a particular point in time diminishes as we move further away from that point.
In short, the AR(p) part describes the overall and long term changes, whereas the MA(q) describes the short-term changes. Together, they allow the ARIMA model to capture a wide range of time series patterns.

Filtering data in R

vindianadoan — Mon, 27 Feb 2023 10:28:49 +0000

Base R

subset() function:

subset(df, column_name > value)

subset operator:

filtered_df <- df[df$column_name > 5, ]

indices in subset operator:

filtered_df <- df[indices, ]

which() function to extract indices:

# create a data frame
df <- data.frame(name = c("Alice", "Bob", "Charlie", "David"),
                 age = c(25, 35, 40, 30))

# use which() to extract indices of rows where age > 30
indices <- which(df$age > 30)

# use the indices to extract the subset of rows
subset_df <- df[indices, ]

# print the resulting data frame
subset_df

Tidyverse/dplyr

filter() function: filter() function filters rows based on certain conditions.

subset_df <- filter(df, column_name > 30)
subset_df <- filter(df, column1_name > 30 
                  & column2_name == TRUE)

slice() function: slice() function selects specific rows.

subset_df <- slice(df, indices)
subset_df <- slice(df, c(1,3))

select() function: select() function selects specific columns.

subset_df <- select(df, column_name)

Why is SSE commonly used in linear regression?

vindianadoan — Sun, 26 Feb 2023 00:37:42 +0000

SSE (sum of squared errors) is commonly used in linear regression because it has some desirable mathematical properties and is relatively easy to optimize using techniques like gradient descent.

One reason SSE is used is because it is a squared error metric. Squaring the errors has a few benefits. First, it makes the errors positive, which means that errors in both directions (i.e., overpredictions and underpredictions) are treated equally. Second, it penalizes large errors more heavily than small errors. This is desirable because we typically care more about reducing large errors than small errors.

Another reason SSE is used is because it leads to a convex optimisation problem. In other words, the cost function has a single global minimum that can be found using gradient descent or other optimization techniques. This is important because it guarantees that the optimisation algorithm will converge to the optimal solution.

Finally, SSE is easy to work with mathematically. The cost function is differentiable, which means that we can calculate its gradient (i.e., the partial derivatives of the cost function with respect to the weights and bias) analytically. This makes it possible to use optimization techniques like gradient descent to find the optimal values of the model parameters.

In addition to SSE, other types of errors used in machine learning include:

Mean Absolute Error (MAE): This measures the absolute differences between the predicted values and the actual values. MAE is less sensitive to outliers compared to SSE.

Root Mean Squared Error (RMSE): This measures the square root of the average of the squared differences between the predicted values and the actual values. RMSE is used when the errors follow a normal distribution.

Mean Absolute Percentage Error (MAPE): This measures the absolute percentage difference between the predicted values and the actual values. MAPE is commonly used in forecasting and time series analysis.

R-squared (R2): This measures the proportion of the variance in the dependent variable that is explained by the independent variables. R2 ranges from 0 to 1, where 1 indicates a perfect fit.

The choice of error metric depends on the type of problem and the nature of the data. Some errors are more appropriate for certain types of data, while others may be more robust to outliers or more intuitive to interpret. It is important to choose the appropriate error metric that aligns with the problem and evaluate the model's performance using that metric.

Supervised learning - Linear regression model

vindianadoan — Sat, 25 Feb 2023 15:29:14 +0000

Linear Regression Model

Concept map:
Linear regression model $→\rightarrow$ SSE $→\rightarrow$ Gradient descent $→\rightarrow$ Training the model

1. Linear Regression Model:
Linear regression is a type of supervised learning algorithm used to predict a continuous target variable based on one or more input features. Other common supervised learning algorithms include logistic regression, decision trees, and neural networks.
The goal of linear regression is to find the best linear relationship between the input features and the target variable. The model takes the form of a linear equation:

y = b_0 + b_1 x_1 + b_2 x_2 + ... + b_n x_n

where:

y

is the target variable

x_1, x_2, ..., x_n

are the input features

b_0

is the y-intercept (also known as the bias)

b_1, b_2, ..., b_n

are the coefficients (also known as weights) of the input features

The objective of linear regression is to find the values of $b_0, b_1, b_2, ..., b_n$ that minimize the difference between the predicted and actual values of $y$ . This is typically done by minimizing the sum of squared errors (SSE) between the predicted and actual values.

2. SSE
By definition:

SSE=∑i=1n(yi−yi^)2SSE = \sum_{i=1}^n (y_i - \hat{y_i})^2

where:

y_i

is the actual value of the target variable for the

i^{th}

data point

yi^\hat{y_i}

is the predicted value of the target variable for the

i^{th}

data point
The minimization of SSE is typically achieved using gradient descent, which is an optimization algorithm that iteratively adjusts the values of

b_0, b_1, b_2, ..., b_n

to find the optimal values that minimize SSE.

3. Gradient Descent:
Gradient descent is an iterative optimization algorithm that is used to find the optimal values of the model parameters (weights and biases) that minimize the cost function (SSE in this case). It works by updating the parameters in the opposite direction of the gradient of the cost function with respect to the parameters.

The update rule for the weights in gradient descent is:

wi=wi−α∂Cost∂wiw_i = w_i - \alpha \frac{\partial Cost}{\partial w_i}

where:

w_i

is the

i^{th}

weight

α\alpha

is the learning rate or step size for each iteration of gradient descent

∂Cost∂wi\frac{\partial Cost}{\partial w_i}

is the partial derivative of the cost function with respect to the

i^{th}

weight
The update rule for the bias term is similar:

b0=b0−α∂Cost∂b0b_0 = b_0 - \alpha \frac{\partial Cost}{\partial b_0}

where:

b_0

is the bias term

∂Cost∂b0\frac{\partial Cost}{\partial b_0}

is the partial derivative of the cost function with respect to the bias term

The partial derivatives of the cost function with respect to the weights and bias can be calculated using calculus.

4. Training the Model:
To train the model, we first initialize the weights and bias to some random values. We then iteratively update the weights and bias using the gradient descent algorithm until the cost function reaches a minimum or a predefined stopping criterion is met (e.g., maximum number of iterations reached).

Once the model is trained, we can use it to make predictions on new data by plugging in the values of the input features into the linear equation and calculating the predicted value of the target variable.

DEV Community: vindianadoan

Fixing AWS CLI Version Conflicts on Windows

Quick Check

Two Fixes

1. Remove the Old CLI

2. Reorder PATH

Setting anaconda Python as the default

Problem

How to Fix It

Method 1: Run Command Prompt as Administrator:

Method 2: Use the Windows Default Apps Settings (No Admin Required):

[Boost]

Part 1: Deploying an R Model as a Containerised API with Docker, Plumber, and renv

vindianadoan ・ Mar 10

Part 1: Deploying an R Model as a Containerised API with Docker, Plumber, and renv

Table of Contents

1. Overview & Prerequisites

2. Building & Saving an R Model

3. Creating a Plumber API Endpoint

4. Building the Base Image with System Dependencies and renv

5. Iterative Model Image for Rapid Development

6. Automating Docker Builds with a Shell Script

7. Managing R Dependencies with renv

8. Testing the API

Running a Container of the Built Image

Testing the Container Using Curl

Testing the Container Using Postman

Testing the Container Using R’s httr Package

9. Troubleshooting

Container fails to install a package

Docker Volume Mount

Interact with Running Container

A cheat sheet on Pandas DataFrame Attributes

1. Core Attributes

2. Data Inspection Attributes

3. Others

ARIMA Modelling: Theory

Introduction

Define the model

Why do we need both an AR component and an MA component?

Filtering data in R

Why is SSE commonly used in linear regression?

Supervised learning - Linear regression model

Linear Regression Model