DEV Community: Hilda Ogamba

Getting Started with AWS SageMaker: Train and Deploy a Model in the Cloud for Cybersecurity Threat Detection (Part 2)

Hilda Ogamba — Tue, 11 Feb 2025 00:50:29 +0000

Welcome Back! 🚀

You’ve made it this far, great job! So far in Part 1, we have:

✅ Part 1 - Set up AWS SageMaker with a Jupyter Notebook instance.
✅ Part 2 - Prepared the NSL-KDD dataset by cleaning, encoding categorical features, and normalizing numerical values and uploaded the dataset to AWS S3 to be used for training.

Now, it’s time to bring everything together!

In this next section, we will go through:
☑️ Part 3 - Train an anomaly detection model on network security data using XGBoost.
☑️ Part 4 - Deploy the trained model as a SageMaker endpoint for real-time threat analysis.
☑️ Part 5 - Make predictions to classify network traffic as normal or malicious.

By the end of this section, you will have a fully deployed cybersecurity threat detection system running in the cloud! Let’s get started. 🚀

Step 3: Train an Anomaly Detection Model on Network Security Data

Now that our preprocessed NSL-KDD dataset is ready in AWS S3, we will:

Use SageMaker’s built-in XGBoost algorithm for binary classification.
Train the model using SageMaker-managed infrastructure.
Store the trained model in S3 for later deployment.

3.1 Set Up SageMaker and Define Training Parameters

First, we will configure the SageMaker environment and specify the XGBoost training job.

import boto3
import sagemaker
from sagemaker import get_execution_role

# Initialize SageMaker session and get execution role
sagemaker_session = sagemaker.Session()
role = get_execution_role()

# Retrieve the built-in SageMaker XGBoost container
region = boto3.Session().region_name
xgboost_image = sagemaker.image_uris.retrieve("xgboost", region, "latest")

print("SageMaker environment configured successfully!")

3.2 Configure the XGBoost Estimator

We now define the XGBoost model, set hyperparameters, and configure the SageMaker training job.

# Define the XGBoost model and training job
xgb_model = sagemaker.estimator.Estimator(
    image_uri=xgboost_image,
    role=role,
    instance_count=1,  # Single instance for training
    instance_type="ml.m5.large",  # Adjust as needed
    volume_size=5,  # Storage in GB for the training container
    output_path=f"s3://{s3_bucket}/{s3_prefix}/output",  # Model output location
    sagemaker_session=sagemaker_session
)

# Set XGBoost hyperparameters
xgb_model.set_hyperparameters(
    objective="binary:logistic",  # Binary classification problem
    num_round=100,  # Number of boosting rounds
    eval_metric="auc",  # Performance metric: Area Under Curve (AUC)
    eta=0.2,  # Learning rate
    max_depth=5,  # Maximum tree depth
    subsample=0.8,  # Use 80% of the data for each boosting round
)

print("XGBoost training job configured successfully!")

3.3 Start Training the Model

We now launch the training job on AWS SageMaker.

# Define SageMaker Training Input
train_input = sagemaker.inputs.TrainingInput(s3_train_data, content_type="csv")

# Start training
xgb_model.fit({"train": train_input})

print("Model training started... ⏳")

Monitor Training Progress

Go to AWS SageMaker Console → Training Jobs to see live logs.
Training time depends on dataset size (usually a few minutes).
Once complete, the trained model will be stored in Amazon S3 at:

s3://cybersecurity-dataset-XXXXX/sagemaker/cybersecurity/output/

Once training completes, we move to Step 4! 🚀

Step 4: Deploy the Model as a SageMaker Endpoint

Now that training is complete, we deploy the trained model as an endpoint so we can send network traffic data for real-time cybersecurity threat detection.

4.1 Deploy the Trained Model

from sagemaker.serializers import CSVSerializer

# Deploy the trained model as a SageMaker endpoint
predictor = xgb_model.deploy(
    initial_instance_count=1,  # Number of deployed instances
    instance_type="ml.m5.large",  # Choose based on expected traffic
    serializer=CSVSerializer(),  # Ensure input data is sent in CSV format
)

print("✅ Model deployed successfully as a SageMaker endpoint!")

✅ SageMaker has now deployed the model! 🎯

Step 5: Make Predictions on Network Traffic Data

Now that the model is deployed, we can send network traffic data to classify it as normal or malicious.

5.1 Test the Model with Sample Data

We will send a normalized network traffic sample to the model for prediction.

import numpy as np

# Example input (Ensure 42 features, replace with real normalized test data)
test_input = np.array([[
    0.1, 0.2, 0.3, 0.5, 0.7, 0.0, 1.0, 0.0, 150, 
    0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 
    0.5, 0.2, 1.0, 0.6, 0.3, 0.1, 0.0, 0.2, 0.3, 0.4, 
    0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.0, 0.3, 0.2, 0.4, 
    0.6, 0.8
]])

print(f"Test input shape: {test_input.shape}")  # Should be (1, 42)

# Send request to the deployed model
response = predictor.predict(test_input.tolist())

# Decode response
probability = float(response.decode("utf-8").strip())  # Convert to float

# Apply threshold to classify as 0 or 1
predicted_label = 1 if probability >= 0.5 else 0

print(f"🔍 Probability Score: {probability:.4f}")
print(f"🔍 Predicted Label: {predicted_label} (0: Normal, 1: Attack)")

5.2 Clean Up Resources

After testing, delete the endpoint to avoid unnecessary AWS charges.

predictor.delete_endpoint()
print("✅ SageMaker endpoint deleted. Training and deployment process complete!")

Remember to stop the notebook instance! Once the instance is stopped, go ahead and delete the it.

Final Thoughts: What We Achieved 🎯

✅ Trained an XGBoost model on network traffic data to detect anomalies.
✅ Deployed the model as a SageMaker endpoint for real-time cybersecurity threat analysis.
✅ Sent test network activity for real-time classification (normal vs. attack).
✅ Shut down resources to avoid unnecessary costs.

🚀 Congratulations! You now have a fully functional, cloud-based cybersecurity anomaly detection system using AWS SageMaker! 🔥

Getting Started with AWS SageMaker: Train and Deploy a Model in the Cloud for Cybersecurity Threat Detection (Part 1)

Hilda Ogamba — Mon, 10 Feb 2025 04:08:24 +0000

Introduction

Why AWS SageMaker?

Cyber threats are growing more sophisticated, and traditional rule-based security systems often fail to detect advanced attacks. Machine Learning (ML) and AWS SageMaker provide a scalable, automated way to analyze large volumes of security logs and detect anomalies in real-time. However, setting up an ML environment can be challenging, requiring:

Powerful compute resources (GPUs, high-memory instances)
Proper data storage and management
Scalability for real-world applications
Model deployment pipelines for making real-time predictions

Enter AWS SageMaker

AWS SageMaker is a fully managed service that simplifies ML by providing:

Pre-configured environments – No need to install ML libraries manually.
Built-in algorithms – Use optimized ML models like XGBoost, TensorFlow, and PyTorch.
Scalability – Train on multiple GPUs or CPUs without managing infrastructure.
Easy Deployment – Deploy models as APIs with a few clicks or lines of code.
Seamless Integration – Works with S3 (for data), Lambda (for automation), and other AWS services.

With AWS SageMaker, security teams can:
✅ Identify suspicious network traffic and insider threats
✅ Detect malware patterns from system logs
✅ Predict potential security breaches before they occur
✅ Automate security response using real-time ML-based alerts

What We’ll Cover in This Guide

In this blog, we’ll take a hands-on approach to AWS SageMaker. We’ll walk through how to train and deploy an ML model on AWS SageMaker to detect cybersecurity threats. We’ll:

Set up AWS SageMaker with a notebook instance 📂
Train an anomaly detection model on network security data 🔍
Deploy the model as an endpoint for real-time threat analysis 🚀
Make predictions to classify normal vs. malicious network activity 🎯

By the end, you’ll have a fully operational cybersecurity threat detection model running in the cloud. Let’s get started!

Step 1: Set Up AWS SageMaker

1.1 Create a SageMaker Notebook Instance

1) Log in to the AWS Management Console and navigate to Amazon SageMaker AI.

2) In the left menu, select Notebook Instances → Click Create notebook instance.

3) Name your instance (e.g., cybersecurity-detection).

4) Choose an instance type (ml.t2.medium for free tier or ml.m5.large for better performance).

5) Create an IAM role with AmazonSageMakerFullAccess and S3FullAccess.

6) Click Create and wait for the instance to be ready (InService).

7) Click Open Jupyter Lab to launch the notebook.

8) Create a new notebook in the JupyterLab view by selecting on the File menu, choose New, and then choose Notebook. Change the name of the file to cybersecurity_detection.ipynb.

9) For Select Kernel, choose conda_python3. This preinstalled environment includes the default Anaconda installation and Python 3.

Step 2: Prepare Cybersecurity Data

For this example, we’ll use a network intrusion detection dataset (like NSL-KDD or CICIDS2017) that contains network traffic labeled as normal or malicious.

2.1 Download the Dataset

In your Jupyter Notebook, run:

import pandas as pd

# Load NSL-KDD dataset (training set)
url = "https://raw.githubusercontent.com/defcom17/NSL_KDD/master/KDDTrain+.txt"
df = pd.read_csv(url, header=None)

# Define column names based on NSL-KDD documentation
column_names = [
    "duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes", "land",
    "wrong_fragment", "urgent", "hot", "num_failed_logins", "logged_in", "num_compromised",
    "root_shell", "su_attempted", "num_root", "num_file_creations", "num_shells",
    "num_access_files", "num_outbound_cmds", "is_host_login", "is_guest_login",
    "count", "srv_count", "serror_rate", "srv_serror_rate", "rerror_rate",
    "srv_rerror_rate", "same_srv_rate", "diff_srv_rate", "srv_diff_host_rate",
    "dst_host_count", "dst_host_srv_count", "dst_host_same_srv_rate",
    "dst_host_diff_srv_rate", "dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate",
    "dst_host_serror_rate", "dst_host_srv_serror_rate", "dst_host_rerror_rate",
    "dst_host_srv_rerror_rate", "label", "difficulty_level"
]

# Assign column names
df.columns = column_names

# Display first few rows
df.head()

2.2 Identify Feature Types

We separate features into continuous (numerical) and symbolic (categorical) types.

1) Symbolic (Categorical) Features:

protocol_type
service
flag
label (attack type, which we will convert to binary)

2) Continuous (Numerical) Features:

All other columns.

2.3 Encode Categorical Features

Since machine learning models cannot process categorical (symbolic) data, we need to convert protocol_type, service, and flag into numerical representations.

from sklearn.preprocessing import LabelEncoder

# Apply label encoding to categorical features
categorical_cols = ["protocol_type", "service", "flag"]
encoder = LabelEncoder()

for col in categorical_cols:
    df[col] = encoder.fit_transform(df[col])

print("Categorical encoding complete!")
df.head()

2.4 Convert Attack Labels into Binary Classes

The label column contains different attack types. To simplify, we will convert it into a binary classification problem:

normal → 0 (Normal Traffic)
Any other attack type (neptune, smurf, etc.) → 1 (Attack)

# Convert attack types into binary labels
df["label"] = df["label"].apply(lambda x: 0 if x == "normal" else 1)

print("Labels converted to binary format (0: Normal, 1: Attack)")
df["label"].value_counts()

2.5 Normalize Continuous Features

Since machine learning models perform better with normalized numerical data, we will scale all continuous features between 0 and 1 using MinMaxScaler.

from sklearn.preprocessing import MinMaxScaler

# Select numerical columns
numerical_cols = [
    "duration", "src_bytes", "dst_bytes", "land", "wrong_fragment", "urgent", "hot",
    "num_failed_logins", "logged_in", "num_compromised", "root_shell", "su_attempted",
    "num_root", "num_file_creations", "num_shells", "num_access_files", "num_outbound_cmds",
    "is_host_login", "is_guest_login", "count", "srv_count", "serror_rate", "srv_serror_rate",
    "rerror_rate", "srv_rerror_rate", "same_srv_rate", "diff_srv_rate", "srv_diff_host_rate",
    "dst_host_count", "dst_host_srv_count", "dst_host_same_srv_rate", "dst_host_diff_srv_rate",
    "dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate", "dst_host_serror_rate",
    "dst_host_srv_serror_rate", "dst_host_rerror_rate", "dst_host_srv_rerror_rate"
]

# Apply MinMax scaling
scaler = MinMaxScaler()
df[numerical_cols] = scaler.fit_transform(df[numerical_cols])

print("Numerical features normalized!")

2.6 Save the Preprocessed Data

Now that the dataset is cleaned, encoded, and normalized, save it for AWS SageMaker.

df.to_csv("cybersecurity_preprocessed.csv", index=False, header=False)
print("Preprocessed data saved!")

2.7 Upload Data to AWS S3

Before training, we need to upload the dataset to AWS S3.

import boto3
import random

# Generate a unique S3 bucket name with 5 random numbers
random_suffix = random.randint(10000, 99999)  # Generate 5-digit random number
s3_bucket = f"cybersecurity-dataset-{random_suffix}"  # Unique bucket name
s3_prefix = "sagemaker/cybersecurity"

# Create an S3 client
s3 = boto3.client("s3")

# Create the S3 bucket (in your default AWS region)
try:
    s3.create_bucket(Bucket=s3_bucket)
    print(f"S3 bucket '{s3_bucket}' created successfully!")
except Exception as e:
    print(f"Error creating S3 bucket: {e}")

# Upload preprocessed dataset to S3
s3.upload_file("cybersecurity_preprocessed.csv", s3_bucket, f"{s3_prefix}/cybersecurity_data.csv")

# Print S3 file path
s3_data_path = f"s3://{s3_bucket}/{s3_prefix}/cybersecurity_data.csv"
print("Data uploaded to S3:", s3_data_path)

Conclusion

So far, we have completed Step 1: Setting up AWS SageMaker and Step 2: Preparing the NSL-KDD dataset for training. Here’s a quick review of our progress and what’s left:

Steps Completed:

✅Step 1: Set Up AWS SageMaker

Launched a SageMaker Notebook Instance in AWS.
Installed necessary dependencies (e.g., boto3, sagemaker).

✅Step 2: Prepare Cybersecurity Data

Loaded and cleaned the NSL-KDD dataset (dropped difficulty_label).
Encoded categorical features (protocol_type, service, flag).
Converted labels to a binary classification format (0: Normal, 1: Attack).
Normalized numerical features using MinMaxScaler.
Saved and uploaded the dataset to an S3 bucket (with unique random digits).

⏳ Next Steps to Complete:

Step 3: Train an Anomaly Detection Model on Network Security Data
Step 4: Deploy the Model as an Endpoint for Real-Time Threat Analysis
Step 5: Make Predictions to Classify Normal vs. Malicious Network Activity

We will complete the rest of this tutorial in Part 2.

Exploring Machine Learning for Credit Card Fraud Detection

Hilda Ogamba — Tue, 03 Dec 2024 23:28:09 +0000

Today, I would like to share some key takeaways from a fascinating project I recently worked on with my team as part of the CIS 635: Knowledge Discovery and Data Mining course, supervised by Professor Zhuang. We tackled a problem that's not only technically challenging but also incredibly relevant: credit card fraud detection. Here's an informal breakdown of what we learned, explored, and achieved during this journey.

Why Fraud Detection?

Machine learning in finance has always intrigued me. With so much at stake in the real world—billions of dollars in losses from fraud annually—I wanted to see if we could design a system that could actually make a difference. When I started this project, I wondered how well our machine learning models could perform on such a critical problem in real-world finance. The rare nature of fraud in datasets posed an added challenge, making this a perfect opportunity to experiment with advanced techniques and evaluate their practical potential.

A Little Disclaimer

Before diving deeper, I should mention that I’m not really a data scientist. My background is more in software engineering and cybersecurity, but I love exploring new challenges. This project gave me a chance to dip my toes into data science concepts like model evaluation, handling class imbalance, and working with algorithms like Random Forest and XGBoost. It’s been a fun learning experience, and I’m excited to share what I’ve learned!

Our Workflow at a Glance

Our approach followed a robust pipeline:

Data Preprocessing: We scaled numerical features (like transaction amounts) and handled the class imbalance using the Synthetic Minority Over-sampling Technique (SMOTE). This allowed us to generate synthetic samples for the minority (fraud) class, enabling more balanced model training.

Model Selection: We tried out several machine learning algorithms, including:
- Logistic Regression (a simple and interpretable baseline)
- Random Forest
- XGBoost (Extreme Gradient Boosting)
- Neural Networks
- Ensemble Learning (combining Random Forest and XGBoost)
Evaluation Metrics: Accuracy, while commonly used, tends to be influenced by the majority class (non-fraudulent transactions), making it a less valuable metric in highly imbalanced datasets like ours. For a problem like fraud detection, metrics such as Precision, Recall, F1-score, and ROC-AUC are more critical than plain accuracy. After all, catching fraud is about finding a balance between false positives (flagging legitimate transactions as fraud) and false negatives (missing actual fraud).
Tools Used: We ran all experiments and analyses on Google Colab, which made it easy to leverage GPU acceleration for training our models. Its collaborative environment also helped streamline our teamwork by allowing us to share and iterate on notebooks efficiently.

Key Highlights from Our Results

Logistic Regression was a great starting point, achieving a ROC-AUC score of 0.9825. However, it struggled with precision, generating too many false positives.

Random Forest offered a balanced performance with a precision of 0.80 and a recall of 0.88, making it a robust choice for real-world deployment.

XGBoost, true to its reputation, slightly outperformed Random Forest with a ROC-AUC score of 0.9911, capturing complex patterns in the data.

Neural Networks excelled in recall (0.90) but suffered from a high false positive rate, making it less ideal for scenarios where precision matters.

Ensemble Learning was the star of the show, combining Random Forest and XGBoost for a ROC-AUC score of 0.9912 and delivering balanced precision (0.80) and recall (0.87).

Comparison of Models

To better illustrate the performance of each model, here's a summary table comparing key metrics like precision, recall, F1-score, ROC-AUC, and AUPRC for each classifier:

Model	Precision	Recall	F1 Score	ROC-AUC	AUPRC	Key Insights
Logistic Regression	0.14	0.93	0.24	0.9825	0.8086	High recall but low precision, leading to many false positives. Suitable for initial screening.
Random Forest	0.80	0.88	0.84	0.9900	0.881	Strong balance of precision and recall with high reliability. Robust for operational environments.
XGBoost	0.78	0.86	0.82	0.9911	0.8858	Slightly better ROC-AUC and AUPRC than Random Forest, effectively captures complex interactions.
Neural Network	0.18	0.90	0.30	0.9684	0.823	High recall but poor precision, leading to frequent false positives.
Ensemble (Voting)	0.80	0.87	0.83	0.9912	0.8812	Combines strengths of Random Forest and XGBoost, achieving a balanced and robust performance.

Comparison table here, showing metrics for Logistic Regression, Random Forest, XGBoost, Neural Networks, and Ensemble Learning.

Challenges We Encountered

Class Imbalance: Fraudulent transactions are rare, and even with SMOTE, striking the right balance without overfitting was tricky.
Interpretable Features: The dataset's anonymized features (due to PCA transformation) limited our ability to interpret the underlying drivers of fraud.
Computational Costs: Hyperparameter tuning across multiple models was resource-intensive. Although Google Colab helped a lot, some configurations took over 7 hours to run!

Why This Matters to Me

This project wasn’t just about fulfilling a course requirement—it was a chance to test how theoretical machine-learning models could hold up in the world of finance. Fraud detection isn't just a fascinating technical problem; it’s also deeply impactful. Testing this on real-world data gave me insights into the nuances of applying ML in high-stakes industries. It was thrilling to see how even basic techniques like Logistic Regression could provide value while advanced models like XGBoost and ensemble methods added an extra layer of sophistication.

Future Directions

Fraud patterns evolve, and our models need to keep up. Here's what we'd love to explore further:

Incorporating unsupervised learning methods for detecting novel fraud patterns.
Using Explainable AI (XAI) to make fraud predictions more transparent and trustworthy.
Enhancing the pipeline for real-time fraud detection by incorporating temporal and geospatial data.

Final Thoughts

This project reaffirmed my belief in the power of collaboration and the importance of domain-specific challenges in shaping technical solutions. Fraud detection is just one application of machine learning where technology can have a real-world impact. Thank you to my teammates, Lynn and Joyce for all your contributions.

I’d love to hear your thoughts! Have you worked on similar problems or encountered challenges with imbalanced datasets? Let’s chat in the comments.

Resources:

Applying the IAMB Algorithm: Feature Selection for Predicting Diabetes in Women

Hilda Ogamba — Thu, 14 Nov 2024 21:35:00 +0000

So, here’s the story—I recently worked on a school assignment by Professor Zhuang involving a pretty cool algorithm called the Incremental Association Markov Blanket (IAMB). Now, I do not have a background in data science or statistics, so this is new territory for me, but I love to learn something new. The goal? Use IAMB to select features in a dataset and see how it impacts the performance of a machine-learning model.

We’ll go over the basics of the IAMB algorithm and apply it to the Pima Indians Diabetes Dataset from Jason Brownlee's datasets. This dataset tracks health data on women and includes whether they have diabetes or not. We’ll use IAMB to figure out which features (like BMI or glucose levels) matter most for predicting diabetes.

What’s the IAMB Algorithm, and Why Use It?

The IAMB algorithm is like a friend who helps you clean up a list of suspects in a mystery—it’s a feature selection method designed to pick out only the variables that truly matter for predicting your target. In this case, the target is whether someone has diabetes.
The IAMB algorithm has two phases:

Forward Phase: Add variables that are strongly related to the target.
Backward Phase: Trim out the variables that don’t really help, ensuring only the most crucial ones are left.

In simpler terms, IAMB helps us avoid clutter in our dataset by selecting only the most relevant features. This is especially handy when you want to keep things simple boost model performance and speed up the training time.

Source: Algorithms for Large-Scale Markov Blanket Discovery

What’s This Alpha Thing, and Why Does it Matter?

Here’s where alpha comes in. In statistics, alpha (α) is the threshold we set to decide what counts as "statistically significant." As part of the instructions given by the professor, I used an alpha of 0.05, meaning I only want to keep features that have less than a 5% chance of being randomly associated with the target variable. So, if a feature’s p-value is less than 0.05, it means there’s a strong, statistically significant association with our target.

By using this alpha threshold, we’re focusing only on the most meaningful variables, ignoring any that don’t pass our “significance” test. It’s like a filter that keeps the most relevant features and tosses out the noise.

Getting Hands-On: Using IAMB on the Pima Indians Diabetes Dataset

Here's the setup: the Pima Indians Diabetes Dataset has health features (blood pressure, age, insulin levels, etc.) and our target, Outcome (whether someone has diabetes).

First, we load the data and check it out:

import pandas as pd
# Load and preview the dataset
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
column_names = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age', 'Outcome']
data = pd.read_csv(url, names=column_names)
print(data.head())

Implementing IAMB with Alpha = 0.05

Here’s our updated version of the IAMB algorithm. We’re using p-values to decide which features to keep, so only those with p-values less than our alpha (0.05) are selected.

import pingouin as pg
def iamb(target, data, alpha=0.05):
    markov_blanket = set()
    # Forward Phase: Add features with a p-value < alpha
    for feature in data.columns:
        if feature != target:
            result = pg.partial_corr(data=data, x=feature, y=target, covar=markov_blanket)
            p_value = result.at[0, 'p-val']
            if p_value < alpha:
                markov_blanket.add(feature)
    # Backward Phase: Remove features with p-value > alpha
    for feature in list(markov_blanket):
        reduced_mb = markov_blanket - {feature}
        result = pg.partial_corr(data=data, x=feature, y=target, covar=reduced_mb)
        p_value = result.at[0, 'p-val']
        if p_value > alpha:
            markov_blanket.remove(feature)
    return list(markov_blanket)

# Apply the updated IAMB function on the Pima dataset
selected_features = iamb('Outcome', data, alpha=0.05)
print("Selected Features:", selected_features)

When I ran this, it gave me a refined list of features that IAMB thought were most closely related to diabetes outcomes. This list helps narrow down the variables we need for building our model.

Selected Features: ['BMI', 'DiabetesPedigreeFunction', 'Pregnancies', 'Glucose']

Testing the Impact of IAMB-Selected Features on Model Performance

Once we have our selected features, the real test compares model performance with all features versus IAMB-selected features. For this, I went with a simple Gaussian Naive Bayes model because it’s straightforward and does well with probabilities (which ties in with the whole Bayesian vibe).

Here’s the code to train and test the model:

from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

# Split data
X = data.drop('Outcome', axis=1)
y = data['Outcome']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Model with All Features
model_all = GaussianNB()
model_all.fit(X_train, y_train)
y_pred_all = model_all.predict(X_test)

# Model with IAMB-Selected Features
X_train_selected = X_train[selected_features]
X_test_selected = X_test[selected_features]

model_iamb = GaussianNB()
model_iamb.fit(X_train_selected, y_train)
y_pred_iamb = model_iamb.predict(X_test_selected)

# Evaluate models
results = {
    'Model': ['All Features', 'IAMB-Selected Features'],
    'Accuracy': [accuracy_score(y_test, y_pred_all), accuracy_score(y_test, y_pred_iamb)],
    'F1 Score': [f1_score(y_test, y_pred_all, average='weighted'), f1_score(y_test, y_pred_iamb, average='weighted')],
    'AUC-ROC': [roc_auc_score(y_test, y_pred_all), roc_auc_score(y_test, y_pred_iamb)]
}

results_df = pd.DataFrame(results)
display(results_df)

Results

Here’s what the comparison looks like:

Using only the IAMB-selected features gave a slight boost in accuracy and other metrics. It’s not a huge jump, but the fact that we’re getting better performance with fewer features is promising. Plus, it means our model isn’t relying on “noise” or irrelevant data.

Key Takeaways

IAMB is great for feature selection: It helps clean up our dataset by focusing only on what really matters for predicting our target.
Less is often more: Sometimes, fewer features give us better results, as we saw here with a small boost in model accuracy.
Learning and experimenting is the fun part: Even without a deep background in data science, diving into projects like this opens up new ways to understand data and machine learning.

I hope this gives a friendly intro to IAMB! If you’re curious, give it a shot—it’s a handy tool in the machine learning toolbox, and you might just see some cool improvements in your own projects.

Source: Algorithms for Large-Scale Markov Blanket Discovery

Connecting your existing EC2 instances to your Amazon EFS file system

Hilda Ogamba — Mon, 06 Mar 2023 16:38:13 +0000

Intro

When you set up a shared EFS file system, it is very easy to mount it onto when we launch an instance. AWS does a lot of the heavy lifting for you, setting up the mount points and enabling automatic mounting whenever you reboot your instance. However, if you want to mount your file system on existing instances, you will have to manually mount the EFS onto your instances. In this step by step tutorial we will demonstrate how you can achieve this using the EFS mount helper.

Prerequisite

For this tutorial, I'll assume you already have an AWS account. If not please create one here.

Steps we will follow

A. First, we will launch two Amazon Linux 2 EC2 instances with SSH access.
B. We will then create an EFS file system.
C. Next, we will SSH into the instances and install the amazon-efs-utils package. We will also create a directory that we will use as the file system mount point.
D. ✨Bonus step✨ We will use EFS mount helper to automatically remount the EFS file system every time we reboot our instances.

Please note that in-order for this setup to function, all of these resources must be launched in the same region.

Let's get started! 🏁

A. Launching Amazon Linux 2 EC2 Instance

Log into the AWS Management Console.
Search for the Amazon EC2 service and open the EC2 console.
Firstly, we'll create two security groups. Choose Security Groups on left-hand panel.
Click Create Security Group.
Under Security Group Name and Description enter Demo EC2 SecGrp.
Under VPC select the default VPC.
For the security group inbound rules, create a new rule that allows SSH(Port 22) and set the source to Anywhere IPv4 (0.0.0.0/0).
We'll keep the outbound rules as default. Once done, choose Create security group.
Once the security group has been created, click Create Security Group again.
Under Security Group Name and Description enter Demo EFS SecGrp.
Under VPC select the default VPC.
For the security group inbound rules, create a new rule with NFS(Port 2049) and set the source to the Demo EC2 SecGrp.
Again, we'll leave the outbound rules as is then choose Create security group.
On the left hand menu, choose Instances.
Next, choose Launch instance.
Under Name, enter the name "Demo" to identify your instance.
Under Application and OS Images (Amazon Machine Image), choose the Amazon Linux 2 AMI.
Under Instance type, select t2.micro.
Under Key pair, for Key pair name, choose an existing key pair or create a new one.
Under Network settings, make sure the default VPC is selected. Under security groups, select from existing security groups and choose the Demo EC2 SecGrp that we previously created.
We can leave all other configurations as is, and then choose Launch instance.

B. Setting Up Amazon EFS

From the search bar type in "Amazon EFS" then select the service from the search results.
On the Amazon EFS dashboard, choose Create file system to open a dialogue box.
Enter the Name "Demo EFS" for your file system.
For Virtual Private Cloud (VPC), choose the same VPC that you launched your instance into.
For Storage class, choose the Standard option.
To customize settings manually, select the Customize option.
We will stick with most of the default service configurations. When you get to the Network access section, clear the default sg from all the Availability Zones and add the EFS SecGrp instead. Click Next.
On all other sections leave the default file systems configurations. Then review the settings and select Create.
Once the file system has been created, click the Demo EFS name entry. You will then be redirected to the File systems page. Choose the Attach option and on the pop up window that appears, copy the script seen under "Using the EFS mount helper" option. It will have the same format as shown below. Save this for later.

sudo mount -t efs -o tls fs-xxxxxx:/ efs

C. Configuring your Linux instance

Open a terminal window and use ssh to connect to the instance.

ssh -i /path/key-pair-name.pem ec2-user@instance-public-ipv4-dns-name

Next, run this command to install the amazon-efs-utils package.
```
sudo yum install -y amazon-efs-utils
```
We now need to create a directory efs that will serve as the file system mount point.
```
sudo mkdir efs
```
Lastly, paste in the command you copied before to mount your file system.
```
sudo mount -t efs -o tls fs-xxxxxx:/ efs
```

Your EC2 instance is now mounted to the EFS file system. However, you'll notice that if you reboot your instance, the file system does not remount to your instance. We can use the /etc/fstab file with EFS mount helper to automatically remount the file system. The /etc/fstab file contains information about file systems that should be mounted during instance booting.
If you run the cat /etc/fstab command on your terminal, you'll notice there are no references to the EFS file system we have mounted. We have to manually update the /etc/fstab file so that the instance uses the EFS mount helper to automatically remount the file system whenever the instance restarts.

D. Setting up Automatic Mounting using `/etc/fstab` with the EFS Mount Helper

Run the following command to open the /etc/fstab file in the nano editor.
```
sudo nano /etc/fstab
```
On a new line, paste in the following (remember to use the file system ID):
```
fs-xxxxxx:/ /home/ec2-user/efs efs tls,_netdev
```
Save the changes to the file by holding down the ctrl and O keys (press ENTER when prompted to save with the file name) then exit the editor by holding down the ctrl and x keys.
Let's test the new entry to see if everything was setup correctly.
```
sudo mount -fav
```

All done! We now have the EC2 instance remounting the file system on reboot.

Guide to Dev.to

Hilda Ogamba — Sat, 14 May 2022 12:46:45 +0000

What is Dev.to

According to the about page, Dev.to is:

DEV is a community of software developers getting together to help one another out. The software industry relies on collaboration and networked learning. We provide a place for that to happen.

Key Features

Sign Up/Sign In

Creating an account lets you bookmark articles you liked and would like to read up on later on and also make your own posts.
You can link other existing accounts to sign up or sign into Dev.to.

Home Page

There are various options on your home page once you sign into your account. The Relevant tab displays various posts that are curated based on the tags, users and organizations you follow or view popular post on the Top tab. The Latest tab has, well, the latest posts.

There are also menu options on the left to navigate the other attributes including:

Listings
Podcasts
Videos
Forem Shop
Sponsors

Posts

Posts are the most popular feature of Dev.to. They can be in a classic format of a blog post where users can interact with the author by posting comments or reactions(likes, bookmarks and unicorn). Posts are also used for discussion threads, a place to host tech content in other media formats, like videos or podcasts. Dev.to uses a markdown editor that uses Jekyll front matter to compose posts. You can read more on the particular formats used on the Editor's Guide Page. You can attach various tags on your posts to help with searches. You can also click on a tag and view all the posts that belong to it!

You can also use RSS feeds to cross-post, allowing people with their own personal blogs and organizations with their own platforms to make new posts on multiple platforms like Medium.

Podcasts

Dev.to also features a host of cool tech podcasts different topics periodically so you can always find something that interests you. You can also make suggestion on podcasts to be added to the selection.

Videos

Dev.to also has a age dedicated to videos uploaded by their community members. These videos are often accompanied by a post meaning that creators can combine multiple media types when making technical content. Videos help augment the user experience. People can follow along with the practical demonstrations which is pretty helpful when creating tutorials for beginners.

Listings

The Listings page features multiple entries organized based on different categories. You can see new events, job listings, offers for mentor-ship, new courses and so much more.

Forem Shop

There is also a bunch of Dev Community Swag available on its Forem Shop. It is a great way to support the platform and to show you are a part of its great Dev community.

Sources

Read more about Dev.to and its features here.

AWS Tools for PowerShell

Hilda Ogamba — Fri, 13 May 2022 14:24:40 +0000

Introduction

From the AWS documentation:

The AWS Tools for PowerShell are a set of PowerShell cmdlets that are built on top of the functionality exposed by the AWS SDK for .NET. The AWS Tools for PowerShell enable you to script operations on your AWS resources from the PowerShell command line.

Just like the AWS CLI, the AWS Tools for PowerShell let developers and administrators manage their AWS services and resources in the PowerShell scripting environment.

Installation and Configuration (on Windows)

These tools are only available from version on computers that are running Windows with Windows PowerShell 5.1, or PowerShell Core 6.0 or later.
Run the Windows PowerShell as Administrator.
Run the following command to install AWS Tools for PowerShell:

Install-Module -Name AWSPowerShell

Answer "Y" to the interactive prompts.

Next, try to see if the application has installed correctly:

Get-AWSPowerShellVersion

Incase you encounter errors stating that "... the module could not be loaded ..." or that the module "... cannot be loaded because running scripts is disabled on this system ...", you may need to change the Execution Policy.
Run the following command to change the Execution Policy:

Set-ExecutionPolicy RemoteSigned

Answer "Y" to the interactive prompts.

Next, set the proper AWS Credentials. If you don't have an administrator set up for your AWS account, follow the instructions here.
Once the administartor user has been set up, generate access keys for that user and securely store that information.

To add a new profile to the AWS SDK store, run the command (replace the AccessKey and SecretKey codes with the administrator access keys):

Set-AWSCredential `
                 -AccessKey AKIA0123456787EXAMPLE `
                 -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY `
                 -StoreAs Administrator

Next, save the configuration as the default profile and region for every PowerShell session:

Initialize-AWSDefaultConfiguration -ProfileName Administrator -Region us-west-2

AWS Tools for PowerShell Example Cmdlets

Here are some example cmdlets:

Create Key-Pair

$myPSKeyPair = New-EC2KeyPair -KeyName myPSKeyPair

View Key-Pair Details

$myPSKeyPair | Get-Member

Save Key-Pair

$myPSKeyPair.KeyMaterial | Out-File -Encoding ascii myPSKeyPair.pem

Remove Key Pair from AWS

Remove-EC2KeyPair -KeyName myPSKeyPair

Create Security Group

New-EC2SecurityGroup -GroupName myPSSecurityGroup -GroupDescription "EC2-Classic from PowerShell"

View Security Groups

Get-EC2SecurityGroup -GroupNames myPSSecurityGroup

Remove Security Group

Remove-EC2SecurityGroup -GroupId sg-0113e926ee099393f

Launch an instance

New-EC2Instance -ImageId ami-0ca285d4c2cda3300 `
    -MinCount 1 `
    -MaxCount 1 `
    -KeyName myPSKeyPair `
    -SecurityGroups myPSSecurityGroup `
    -InstanceType t2.micro

View Instance

$reservation = New-Object 'collections.generic.list[string]'
$reservation.add("r-b70a0ef1")
$filter_reservation = New-Object Amazon.EC2.Model.Filter -Property @{Name = "reservation-id"; Values = $reservation}
(Get-EC2Instance -Filter $filter_reservation).Instances

Terminate Instance

Remove-EC2Instance -InstanceId i-0453116cdface121d

List Buckets

Get-S3Bucket

Create a bucket

New-S3Bucket -BucketName uniquebucketname2258 -Region us-west-2

Upload an object to the bucket

Write-S3Object -BucketName uniquebucketname2258 -File .\sample.txt

List Bucket Contents

Get-S3Object -BucketName uniquebucketname2258

Delete Bucket and its objects

Remove-S3Bucket -BucketName uniquebucketname2258 -DeleteBucketContent

List Users

Get-IAMUserList

Create a user

New-IAMUser -UserName Bob

Create a new group

New-IAMGroup -GroupName DummyGroup

Add users to group

Add-IAMUserToGroup -UserName "Bob" -GroupName "DummyGroup"

List the group, group details and its users

$results = Get-IAMGroup -GroupName "DummyGroup"
$results
$results.Group
$results.Users

Removes all users from a group and then deletes the group

(Get-IAMGroup -GroupName DummyGroup).Users | Remove-IAMUserFromGroup -GroupName DummyGroup -Force
Remove-IAMGroup -GroupName DummyGroup -Force

Deletes a user

Remove-IAMUser -UserName Bob

For more information on AWS Tools for PowerShell visit AWS Tools for PowerShell Cmdlet Reference page

Infrastructure as Code: AWS CloudFormation

Hilda Ogamba — Mon, 24 Jan 2022 16:01:55 +0000

Introduction

Infrastructure as code (IaC) is the process of provisioning and managing your cloud computing services by coming up with a template file. This lets you reliably and efficiently replicate environment architecture on-demand. AWS CloudFormation is the built-in IaC service. To provision these resources on AWS, you can describe them in a text file with either a .json, .yaml, .template, or .txt extension.

Prerequisites

You will need an active AWS account. Please note that it is advised to follow security guidelines when creating an AWS account to prevent unauthorized access. To follow along from the terminal/command line, you can follow these steps to install AWS CLI and quickly configure the basic settings on your computer.

Benefits of AWS CloudFormation

An infrastructure as code template acts as a clear record of resources that you have provisioned.
If you accidentally change the the wrong settings while deploying resources, it may break things. IaC helps solve this, especially when it is combined with version control, such as Git.
IaC is also reusable meaning that one comprehensive template can be utilized multiple times, in multiple regions, making it much easier to horizontally scale.
By creating IaC that follows proper security guidelines you can reuse it with the assurance that the resource settings are secure.
CloudFormation can verify that the architecture was successfully deployed, and if there is a failure it can gracefully roll-back the infrastructure to a previously stable state.

AWS CloudFormation concepts

There are three main concepts in CloudFormation:

a) Template: This is the text file that describes your architecture. The template is then used to deploy these resources either through the CLI or the console.
Here is an example of a YAML template. This describes a VPC, one public subnet, an EC2 instance (running a web server) and other dependent resources:

AWSTemplateFormatVersion: 2010-09-09

Description: Template example

# VPC with public subnet and Internet Gateway

Parameters:

  ExampleVpcCidr:
    Type: String
    Default: 10.0.0.0/20

  PublicSubnetCidr:
    Type: String
    Default: 10.0.0.0/24

  AmazonLinuxAMIID:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2


Resources:

###########
# VPC with Internet Gateway
###########

  ExampleVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref ExampleVpcCidr
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: Example VPC

  IGW:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: Example IGW

  VPCtoIGWConnection:
    Type: AWS::EC2::VPCGatewayAttachment
    DependsOn:
      - IGW
      - ExampleVPC
    Properties:
      InternetGatewayId: !Ref IGW
      VpcId: !Ref ExampleVPC

###########
# Public Route Table
###########

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    DependsOn: ExampleVPC
    Properties:
      VpcId: !Ref ExampleVPC
      Tags:
        - Key: Name
          Value: Public Route Table

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn:
      - PublicRouteTable
      - IGW
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref IGW
      RouteTableId: !Ref PublicRouteTable

###########
# Public Subnet
###########

  PublicSubnet:
    Type: AWS::EC2::Subnet
    DependsOn: ExampleVPC
    Properties:
      VpcId: !Ref ExampleVPC
      MapPublicIpOnLaunch: true
      CidrBlock: !Ref PublicSubnetCidr
      AvailabilityZone: !Select 
        - 0
        - !GetAZs 
          Ref: AWS::Region
      Tags:
        - Key: Name
          Value: Public Subnet

  PublicRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    DependsOn:
      - PublicRouteTable
      - PublicSubnet
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet


###########
# App Security Group
###########

  AppSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    DependsOn: ExampleVPC
    Properties:
      GroupName: App
      GroupDescription: Enable access to App
      VpcId: !Ref ExampleVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: App

###########
# EC2 Instance
###########

  Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: !Ref AmazonLinuxAMIID
      SubnetId: !Ref PublicSubnet
      SecurityGroupIds:
        - !Ref AppSecurityGroup
      Tags:
        - Key: Name
          Value: App Server
      UserData:
        Fn::Base64: |
          #!/bin/bash
          yum update -y
          yum install -y httpd.x86_64
          systemctl start httpd.service
          systemctl enable httpd.service
          echo “Hello from $(hostname -f)” > /var/www/html/index.html

###########
# Outputs
###########

Outputs:

  PublicDnsName:
    Description: EC2 Instance Public DNS Name
    Value: !GetAtt Instance.PublicDnsName

This particular template is divided into 5 sections:

AWSTemplateFormatVersion: This is a string that describes the AWS CloudFormation template version that the template conforms to. It is denoted as a "version date" format.
Description: It is a string that gives a short summary of the content of the template.
Parameters: It contains the values that are passed at runtime.
Resources: This is the most important (and compulsory) section on a template. It basically describes all the resources to be deployed and their properties.
Outputs: These are values that are returned once a stack is created or updated. The outputs can be viewed from the AWS console. Or by running this command from your terminal/command line:

aws cloudformation describe-stacks --stack-name mystack --query "Stacks[0].Outputs" --output text

In this example, the output returns the public DNS name of the EC2 instance. You can copy this DNS name to your browser and it should direct you to a simple web page.

b) Stack: Once you upload this template, these resources are deployed as a single unit referred to as a stack. You can make changes to resources in a stack and in-case of any failures, these changes can be rolled-back. You can also delete a stack and therefore all the resources in a stack. When creating a new stack, make sure give each stack a unique name. You can watch a video tutorial on how to deploy a template from the AWS console. Or you can upload it through the AWS CLI:

aws cloudformation create-stack --stack-name mystack --template-url http://examplebucketname.s3.amazonaws.com/exampletemplate.yaml

You can delete the stack by running this from AWS CLI:

aws cloudformation delete-stack --stack-name mystack

c) Change Set: When updating a resource, you can simply upload an updated template and let CloudFormation make the necessary changes. This can be risky especially if the changes may delete or modify critical resources. A safer alternative is to use change sets. Change sets allow you to preview the proposed modifications to your current architecture before implementing them. We'll make a modification to our template switching the EC2 instance type from t2.micro to t2.small:

###########
# EC2 Instance
###########

  Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.small
      .
      .
      .

You can watch a video tutorial on how to deploy a change set from the AWS console. Or you can upload the change stack through the CLI:

aws cloudformation create-change-set --stack-name mystack --change-set-name changeset-update --template-url http://examplebucketname.s3.amazonaws.com/updatedexampletemplate.yaml

To view the changes to be implemented in a change set, run this from the terminal:

aws cloudformation describe-change-set --stack-name mystack --change-set-name changeset-update

To execute a change set, run this command in the AWS CLI:

aws cloudformation execute-change-set --stack-name mystack --change-set-name changeset-update

Conclusion

Hope this helps you get started!
PS: Don't forget to delete your stack once you are through to avoid extra charges.

Deploy your React projects to AWS Elastic Beanstalk using CI/CD AWS CodePipeline (Part 2)

Hilda Ogamba — Wed, 19 Jan 2022 17:46:15 +0000

Introduction

In Part 1, we created a React application and uploaded it to a GitHub Repo. We also created an Elastic Beanstalk application. Now, we will pick up where we left off and create a continuous integration/continuous deployment pipeline using CodePipeline.

Create a pipeline

Type 'codepipeline' into the search bar. Select CodePipeline:

Then, click the Create pipeline button:

Type in a name for your pipeline. Leave everything else as it is, then click next:

Next, we will select the code source. Choose 'GitHub (Version 1)' for this tutorial. Click the Connect to GitHub button:

You will be prompted to authorize a AWS CodePipeline connection:

Afterwards, confirm the new configurations made:

Choose the react-demo repo and the branch main from the drop-down menus. Then click 'Next':

Skip the build stage:

In the deployment stage, select the deploy provider as Elastic Beanstalk. Select the region where you launched the Elastic Beanstalk application. Choose the appropriate application name and environment:

Review the configurations, then click Create pipeline:

It will take a couple of minutes for your pipeline to finish setting up and deploy your application. You should see a success message once it's complete:

Navigate back to the Elastic Beanstalk application:

Click this link and it will redirect you to the deployed React application:

Now, we'll make a small change to the application and we'll see the changes reflected on the website. Make a change to your local repo and push it to the GitHub repo:

git add .
git commit -m "Update React application"
git push -u origin main

In a couple of minutes, the website successfully updates:

Congrats, you have successfully setup an automated continuous deployment and continuous integration pipeline. You can continue to make changes to your application and watch them get rolled out in near real-time.

Happy Coding!

Deploy your React projects to AWS Elastic Beanstalk using CI/CD AWS CodePipeline (Part 1)

Hilda Ogamba — Wed, 19 Jan 2022 13:10:54 +0000

Introduction

AWS offers a wide range of on-demand cloud services. This can be very intimidating for beginners that are new to cloud deployment services and those unfamiliar with the AWS infrastructure. That's where AWS Elastic Beanstalk comes in. AWS Elastic Beanstalk is a service that lets you quickly deploy applications in the AWS Cloud without worrying about the underlying infrastructure that those applications run on. All you have to do is upload your application files, and AWS Elastic Beanstalk handles the rest. Simple, right? Well, what if you want to make changes to your application later on? How would these changes be deployed rapidly and efficiently? A great tool for this would be AWS CodePipeline. AWS CodePipeline automates the continuous delivery process and it also integrates with third-party services such as GitHub (where the React Repo for this project is hosted). This will allow us to set up a Continuous Integration and Continuous Delivery (CI/CD) AWS pipeline. Let's get started!

Prerequisites

You will need an active AWS account and GitHub account (or Bit Bucket). Please note that it is advised to follow security guidelines when creating an AWS account to prevent unauthorized access. For this project, it's required that you have Node.js installed in your computer.

Create the React Application

From your terminal/command line, move to the directory of your choice:

cd Desktop

Then, create a React application using the create-react-app tool:

npx create-react-app react-demo

Once the install is completed, change directory to your new application:

cd react-demo

Start your React application:

npm start

This command will start up the Node.js server and launch a new browser window displaying your app. You can use ctrl + c from the terminal/command line to stop running the React app.

Create GitHub Repo

From your browser, navigate to your GitHub account and create a new repo:

Next, follow the instructions to push an existing repository from the command line. They will look similar to this:

Create an Elastic Beanstalk application

Sign into your AWS account. On the home page, type 'elastic beanstalk' into the search bar. Select Elastic Beanstalk:

Next, click the Create Application button:

Give you application a name. I used the name react-demo-app:

Under the 'Platform' section, select the platform as Node.js. Leave everything else at their default settings and click Create Application:

An environment was automatically created for the new application since I had no other existing environments. In my case, the environment name was 'Reactdemoapp-env'. It takes a few minutes to get everything running so we can go ahead and create our pipeline in Part 2.

RF Encrypted Communication System

Hilda Ogamba — Thu, 15 Apr 2021 15:59:45 +0000

In this tutorial, we'll go through how you can send encrypted data from one Arduino to another. The receiver must then enter the correct password to view the decrypted message.

We will work with the Arduino Uno development board, though any equivalent microprocessor will do.

The 433MHz RF module kit is a transmitter-receiver pair (sometimes packaged with an antenna but any standard 17cm wire will work just fine) that sends information one-way. Read more about the RF module here.
There are other long-range radio communication tools out there, so feel free to swap out the RF module for something better.

What you will need

Component	Quantity
Arduino Uno	2
433 MHz RF Receiver Module	1 (Transmitter-Receiver Pair)
20x4 LCD with I2C	1
4x3 Keypad	1
Breadboard	1
Jumper wires	1
(Optional) 17cm Antenna	2

(Optional) Solder on the antennas

You may need to solder the antennas onto the RF module to improve the coverage if you want to transmit over longer distances. For prototyping purposes, you may skip this step.

Installing the Software

You will need to install the Arduino IDE to upload sketches to the Arduino boards and the Processing Development Environment for the GUI. You will also need to download the RadioHead Library and include it in the Arduino libraries.

Finally, download this AESLib Arduino encryption library.

So let's get started!

Setup

This is the transmitter setup.

This is the circuit I implemented.

This is the receiver setup.

This is the implemented circuit.

The code

First, let's start with Processing. Open a new file and copy the code below and run it to create the user interface. We will use the GUI to send data to the transmitter circuit instead of writing messages and the encryption key directly to the serial.

// Import ControlP5, Serial, Regex libraries
import processing.serial.*;
import controlP5.*;
import java.util.regex.*;
// Create Serial object
Serial port;
// Create ControlP5 object
ControlP5 controlP5;
// Create Textfield objects
Textfield messageField;
Textfield keyField;
// Create Textarea objects
Textarea errorField;
Textarea cipherText;
// Create PFont objects
PFont font;
PFont smallerFont;

String encryptionKey;
String message;
String errorMessage;
String dataSent;
String inBuffer = null;

int lf = 10; // Linefeed in ASCII

// Setup the processing programme
void setup() {
    size(700, 800); // Window size (width, height)
    font = createFont("Calibri", 30); // Set font type, sizes
    smallerFont = createFont("Calibri", 20);

    try {
        port = new Serial(this, Serial.list()[0], 57600); // Set Serial to first availble port, baudrate 57600
        port.clear(); // Clear serial
        inBuffer = port.readStringUntil(lf); // Throw out the first reading, in case we started reading 
        inBuffer = null; // in the middle of a string from the sender.
    } catch(Exception e) { // In case of unavailable Serial port
        errorMessage = "Please connect the device to the USB port and relaunch the application";
        println(errorMessage); // Print error message and error type in the Terminal
        println(e);
    }

    controlP5 = new ControlP5(this);

    keyField = controlP5
       .addTextfield("key") // Set name of textfield in quotes
       .setPosition(100, 100) // Set position (x, y)
       .setSize(500, 50) // Set size (width, height)
       .setFont(font) // Set font
       .setAutoClear(false); // Remove auto clear on ENTER
    messageField = controlP5
       .addTextfield("message") // Set name of textfield in quotes
       .setPosition(100, 200) // Set position (x, y)
       .setSize(500, 50) // Set size (width, height)
       .setFont(font) // Set font
       .setAutoClear(false); // Remove auto clear on ENTER
    // Set textareas
    errorField = controlP5
       .addTextarea("error") // Set name of textarea in quotes
       .setPosition(100, 300) // Set position (x, y)
       .setSize(500, 50) // Set size (width, height)
       .setFont(smallerFont); // Set font
    cipherText = controlP5
       .addTextarea("ciphertext") // Set name of textarea in quotes
       .setPosition(100, 600) // Set position (x, y)
       .setSize(500, 100) // Set size (width, height)
       .setFont(smallerFont) // Set font
       .setColorBackground(color(21,27,84));
    // Set label for textarea "cihertext"
    controlP5.addTextlabel("cipherLabel") // Set name of textlabel in quotes
       .setPosition(100, 700) // Set position (x, y)
       .setSize(500, 50) // Set size (width, height)
       .setFont(font) // Set font
       .setValue("CIPHERTEXT"); // Set textlabel value in quotes
    // Set buttons which call functions Send, Clear when clicked
    controlP5.addButton("Send") // Set name of button in quotes
       .setPosition(300, 400) // Set position (x, y)
       .setSize(100, 50) // Set size (width, height)
       .setFont(font); // Set font
    controlP5.addButton("Clear") // Set name of button in quotes
       .setPosition(300, 500) // Set position (x, y)
       .setSize(100, 50) // Set size (width, height)
       .setFont(font); // Set font
}

void draw() {
    try {
        background(180);
        while(port.available() > 0) {
            inBuffer = port.readStringUntil(lf); // Read string from buffer until new line feed   
            if (inBuffer != null) {
                println(inBuffer); // Display encrypted message that is currently transmitting
                cipherText.setText(inBuffer);
            }
        }       
    } catch(Exception e) { // In case of unavailable Serial port
        errorMessage = "Please connect the device to the USB port and relaunch the application";
        errorField.setText(errorMessage); // Display error message in error field
    }
}
//Process message to be sent
void Send() {
    encryptionKey = keyField.getText(); // Set strings from the textfields entries
    message = messageField.getText();
    // Check that the key, message are 16 characters long
    if (encryptionKey.length() == 16 && message.length() == 16) {
        // Check that the encryption key only contains 16 numbers
        if (Pattern.matches("[0-9]{16}", encryptionKey)) {
            dataSent = encryptionKey + "|" + message; // Data sent has the separator "|"
            port.write(dataSent);
            errorField.clear();
        } else { // Error message
            errorMessage = "Key must be an unsigned whole number";
            errorField.setText(errorMessage);
        }    
    } else { // Error message
        errorMessage = "Key must be 16 characters long & Message must be 16 characters long";
        errorField.setText(errorMessage);
    }
}
//Clear the key,message and error fields
void Clear() {
    keyField.clear();
    messageField.clear();
    errorField.clear();
}

Next, we'll work on the transmitter setup. Create a new Arduino sketch. Make sure you extract the AESLib files into the same folder as the Arduino sketch. Copy the code into the Arduino sketch and upload it to the transmitter Arduino Uno.

// Include AES Encryption Library
#include "AESLib.h"
// Include RadioHead Amplitude Shift Keying Library
#include <RH_ASK.h>
// Include dependant SPI Library
#include <SPI.h>

// Create Amplitude Shift Keying Object
RH_ASK rf_driver;

void setup() {
  // put your setup code here, to run once:
  // Initialize ASK Object
  rf_driver.init();
  // Setup Serial Monitor
  Serial.begin(57600);
}

void loop() {
  // put your main code here, to run repeatedly:
  uint8_t key[17]; // 16 characters + 1 for null character
  char text[17]; // 16 characters + 1 for null character

  if (Serial.available() > 0) {
    String msgString = Serial.readString(); // Read data string
    String strkey;
    String message;

    // Split string into two values
    for (int i = 0; i < msgString.length(); i++) {
      if (msgString.substring(i, i + 1) == "|") {
        strkey = msgString.substring(0, i);
        message = msgString.substring(i + 1);

        break;
      }
    }
    // Assign values to key, text
    strkey.toCharArray(key, strkey.length() + 1);
    message.toCharArray(text, message.length() + 1);
    // Encrypt message using the key with AES-128 ECB mode
    aes128_enc_single(key, text);
  }
  // Send output character
  rf_driver.send((uint8_t *)text, strlen(text));
  rf_driver.waitPacketSent();
  // Print encrypted text to Serial Monitor
  Serial.print("encrypted:");
  Serial.println(text);
  delay(1000);
}

Lastly, create another Arduino sketch and copy the code below. Once again, extract the AESLib files into the folder containing the Arduino sketch. Upload this sketch on the receiver Arduino Uno.

// Include Wire Library for I2C
#include <Wire.h>
// Include NewLiquidCrystal Library for I2C
#include <LiquidCrystal_I2C.h>
// Include RadioHead Amplitude Shift Keying Library
#include <RH_ASK.h>
// Include dependant SPI Library
#include <SPI.h>
// Include the Keypad library
#include <Keypad.h>
// Include AES Encryption Library
#include "AESLib.h"

// Length of password + 1 for null character
#define Password_Length 17
// Character to hold password input
char Data[Password_Length];
// Counter for character entries
byte data_count = 0;
// Character to hold key input
char customKey;

// Constants for row and column sizes
const byte ROWS = 4;
const byte COLS = 3;

// Array to represent keys on keypad
char hexaKeys[ROWS][COLS] = {
  {'1', '2', '3'},
  {'4', '5', '6'},
  {'7', '8', '9'},
  {'*', '0', '#'}
};

// Connections to Arduino
byte rowPins[ROWS] = {9, 8, 7, 6};
byte colPins[COLS] = {5, 4, 3};

// Create keypad object
Keypad customKeypad = Keypad(makeKeymap(hexaKeys), rowPins, colPins, ROWS, COLS);

// Define LCD pinout
const int  en = 2, rw = 1, rs = 0, d4 = 4, d5 = 5, d6 = 6, d7 = 7, bl = 3;

// Define I2C Address - change if reqiuired
const int i2c_addr = 0x3F;

LiquidCrystal_I2C lcd(i2c_addr, en, rw, rs, d4, d5, d6, d7, bl, POSITIVE);

// Create Amplitude Shift Keying Object
RH_ASK rf_driver;

void setup() {
  // put your setup code here, to run once:
  // Initialize ASK Object
  rf_driver.init();
  // Set display type as 20 char, 4 rows
  lcd.begin(20, 4);
  //  Setup Serial Monitor
  Serial.begin(57600);
  // Print on first row
  lcd.setCursor(0, 0);
  lcd.print("Up and Running!");

  // Wait 1 second
  delay(1000);

  // Print on second row
  lcd.setCursor(0, 1);
  lcd.print("Setting up ...");

  // Wait 4 seconds
  delay(4000);

  // Clear the display
  lcd.clear();
}

void loop() {
  // put your main code here, to run repeatedly:
  // Set buffer to size of expected message
  uint8_t buf[16];
  uint8_t buflen = sizeof(buf);
  // Initialize LCD and print
  lcd.setCursor(0, 0);
  lcd.print("Enter Password:");
  // Look for keypress
  customKey = customKeypad.getKey();
  if (customKey) {
    // Enter keypress into array and increment counter
    Data[data_count] = customKey;
    lcd.setCursor(data_count, 1);
    lcd.print(Data[data_count]);
    data_count++;
  }

  if (customKey == '*') {
    // '*' keypress to clear the LCD display, Data
    clearData();
  }

  // See if we have reached the password length
  if (data_count == Password_Length - 1) {
    delay(1000);
    lcd.setCursor(0, 3);
    lcd.print("Processing ...");
    // Check if received packet is correct size
    if (rf_driver.recv(buf, &buflen)) {
      uint8_t * key = (uint8_t *)Data;
      char * text = (char *)buf;
      Serial.print("encrypted:");
      Serial.println(text);
      Serial.print("key:");
      Serial.println((char *)key);
      // Message received is decrypted, displayed
      aes128_dec_single(key, text);
      Serial.print("decrypted:");
      Serial.println(text);
      // Clear the LCD display
      lcd.clear();
      lcd.setCursor(0, 0);
      // Display decrypted text
      lcd.print("Message Received: ");
      lcd.setCursor(0, 1);
      lcd.print(text);
      // Wait 30 seconds
      delay(30000);
      // Clear data and LCD display
      clearData();
    }
  } else if (data_count >= Password_Length) {
    // Clear LCD dispaly, Data if password exceeds 16 characters
    clearData();
  }
}

void clearData() {
  // Clear LCD display
  lcd.clear();

  // Go through array and clear data
  while (data_count != 0) {
    Data[data_count--] = 0;
  }
  return;
}

Demonstration

Connect the transmitter setup to a USB port and run the Processing sketch. Type in your own 16-character long message and password and click send.

Connect the receiver setup to a USB port and open the serial monitor. Make sure it is connected to the right port with the baud rate set at 57600. Enter the password when prompted.

Note: The message will only persist for 30 seconds before prompting you to enter a password again.

Wrapping up

An encrypted RF Communication System can be pretty useful when trying to limit access to a broadcasted message. Although the RF 433MHz module is limited in range, it is a fairly inexpensive way to implement wireless connectivity between two IoT devices.

Happy Coding!

DEV Community: Hilda Ogamba

Getting Started with AWS SageMaker: Train and Deploy a Model in the Cloud for Cybersecurity Threat Detection (Part 2)

Welcome Back! 🚀

Step 3: Train an Anomaly Detection Model on Network Security Data

3.1 Set Up SageMaker and Define Training Parameters

3.2 Configure the XGBoost Estimator

3.3 Start Training the Model

Monitor Training Progress

Step 4: Deploy the Model as a SageMaker Endpoint

4.1 Deploy the Trained Model

Step 5: Make Predictions on Network Traffic Data

5.1 Test the Model with Sample Data

5.2 Clean Up Resources

Final Thoughts: What We Achieved 🎯

Getting Started with AWS SageMaker: Train and Deploy a Model in the Cloud for Cybersecurity Threat Detection (Part 1)

Introduction

Why AWS SageMaker?

Enter AWS SageMaker

What We’ll Cover in This Guide

Step 1: Set Up AWS SageMaker

1.1 Create a SageMaker Notebook Instance

Step 2: Prepare Cybersecurity Data

2.1 Download the Dataset

2.2 Identify Feature Types

2.3 Encode Categorical Features

2.4 Convert Attack Labels into Binary Classes

2.5 Normalize Continuous Features

2.6 Save the Preprocessed Data

2.7 Upload Data to AWS S3

Conclusion

Exploring Machine Learning for Credit Card Fraud Detection

Why Fraud Detection?

A Little Disclaimer

Our Workflow at a Glance

Key Highlights from Our Results

Comparison of Models

Challenges We Encountered

Why This Matters to Me

Future Directions

Final Thoughts

Applying the IAMB Algorithm: Feature Selection for Predicting Diabetes in Women

What’s the IAMB Algorithm, and Why Use It?

What’s This Alpha Thing, and Why Does it Matter?

Getting Hands-On: Using IAMB on the Pima Indians Diabetes Dataset

Implementing IAMB with Alpha = 0.05

Testing the Impact of IAMB-Selected Features on Model Performance

Results

Key Takeaways

Connecting your existing EC2 instances to your Amazon EFS file system

Intro

Prerequisite

Steps we will follow

A. Launching Amazon Linux 2 EC2 Instance

B. Setting Up Amazon EFS

C. Configuring your Linux instance

D. Setting up Automatic Mounting using /etc/fstab with the EFS Mount Helper

Guide to Dev.to

What is Dev.to

Key Features

Sign Up/Sign In

Home Page

Posts

Podcasts

Videos

Listings

Forem Shop

Sources

AWS Tools for PowerShell

Introduction

Installation and Configuration (on Windows)

AWS Tools for PowerShell Example Cmdlets

Create Key-Pair

View Key-Pair Details

Save Key-Pair

Remove Key Pair from AWS

Create Security Group

View Security Groups

Remove Security Group

Launch an instance

View Instance

D. Setting up Automatic Mounting using `/etc/fstab` with the EFS Mount Helper