DEV Community: Aldo Portillo

Load Balancer Algorithms

Aldo Portillo — Wed, 04 Sep 2024 15:30:35 +0000

Introduction

Before I decided to switch to AWS for my personal projects, I studied system design. When studying system design the diagrams were slightly difficult to follow. I wanted more visuals, so I booted up Nginx and created a simple load balancer to help me understand the different algorithms through logs. This seemed like a huge hurdle just to see how each algorithm works, so I decided to create an interactive demo to help others understand load balancer algorithms.

Load balancers are used to distribute incoming network traffic across multiple servers. This is done to ensure that no single server is overwhelmed with requests. Load balancers can be implemented in software or hardware and can be used to distribute traffic across multiple servers, data centers, or even cloud providers.

Static Load Balancing

Static load balancing algorithms distribute traffic evenly across a set of servers. These algorithms do not take into account the current load on each server and are not able to adapt to changes in traffic patterns.

Round Robin

Round Robin is one of the simplest load balancing algorithms. It distributes incoming requests to a group of servers in a circular order. Each server is assigned a request in turn, and the cycle repeats.

Weighted Round Robin

Weighted Round Robin is an extension of the Round Robin algorithm that assigns a weight to each server. Servers with higher weights receive more requests per cycle than servers with lower weights. You can adjust the weights and cycle in the demo below to distribute traffic according to the capacity of each server.

Hashed Load Balancing

Hashed Load Balancing uses a hash function to hash an IP address an map incoming requests to a specific server. This ensures that requests for the same resource are always directed to the same server. Directing requests to the same server can be useful for caching or session management if they are performed on a per-server basis.

Demo

The demo is only available on my portfolio. Feel free to check it out here. I will soon add dynamic load balancing algorithms.

Fetch External APIs in Next.js

Aldo Portillo — Wed, 04 Sep 2024 15:27:19 +0000

Introduction

Next.js is a up and coming framework for building React applications. It focuses on Server side rendered (SSR) applications to provide better search engine optimization (SEO) and performance. Why Next.js? Next is briding the gap between single page applications (SPA) and SSR applications. This is a fairly new concept, so as expected, there are some challenges that come with it. In October 2023, Next.js version 14 was released and it seemed as if an entire new framework was introduced when they switched from the /pages router to /app router. This made their previous documentation outdated. Since I didn't adapt Next before this change, I had to learn from the new documentation. Unfortunately, the new documentation was not as detailed as the previous one and it referenced the previous one a lot. This made it difficult to understand how to fetch external APIs in Next.js. In this post, I will share how to make an external API request in Next and share one of my favorite form security strategies.

Fetching External APIs in Next.js

You might be thinking to yourself: "Aldo why is it that you don't just use a client component?" Well, I am using an environment variable that I don't want to expose to the client.

//In /app/api/sendContactEmail/route.js
const postmark = require('postmark');

const client = new postmark.ServerClient(process.env.POSTMARK_KEY);

export async function POST(request) {
    const { email, name, message, honey } = await request.json();

    if (honey) {
        console.error("Honey pot field was filled out");
        return new Response(JSON.stringify({ error: "Honey pot field was filled out" }), {
            status: 400,
            headers: { 'Content-Type': 'application/json' }
        });
    }

    const templateModel = {
        email: email,
        name: name,
        message: message
    };

    try {
        const result = await client.sendEmailWithTemplate({
            //More Postmark API options here
            "TemplateModel": templateModel
        });

        console.info("Sent to postmark for delivery", result);
        return new Response(JSON.stringify({ message: 'Email sent successfully' }), {
            status: 200,
            headers: { 'Content-Type': 'application/json' }
        });
    } catch (error) {
        console.error("Unable to send via postmark: " + error.message);
        return new Response(JSON.stringify({ error: error.message }), {
            status: 500,
            headers: { 'Content-Type': 'application/json' }
        });
    }
}

In the code above, I am using the Postmark API to send an email. I am using the process.env.POSTMARK_KEY to authenticate with Postmark. This is an environment variable that I don't want to expose to the client. This is why I am using a server component to make the request. Take a note of the directory. We have this code inside of /app/api/sendContactEmail/route.js. This is because we are using the new /app router. Similar to page.js, route.js is a reserved file name.

If you run your build command you will see that this route is dynamic.

$ npm run build

Route (app)                              Size     First Load JS
┌ ○ /                                    375 B          99.5 kB
├ ○ /_not-found                          875 B          88.1 kB
├ ƒ /api/sendContactEmail                0 B                0 B
├ ○ /blogs                               323 B          94.4 kB
├ ƒ /blogs/[blog]                        1.25 kB        88.5 kB

Ignore the large sizes. I'm still learning Next.js.

In your component you can now call this route like so:

//In /app/pages/contact.js
"use client"
export default function Contact() {
    //Set States and Effects here

    const handleSubmit = async (e) => {
        e.preventDefault();
        const response = await fetch('/api/sendContactEmail', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({ email, name, message, honey })
        });

        const data = await response.json();
        console.log(data);
    }

    return (
        //Form here
    );
}

Yes we are in a client component; however, the fetch is being made to a server component.

Form Security

In the code above, I am using a honey pot field to prevent spam. A honey pot field is a hidden field that is only visible to bots. If the field is filled out, then we know that it is a bot. This is a simple and effective way to prevent spam. I am doing this because I don't want to use a captcha. I find captchas to be annoying and I don't want to annoy my users, but I also don't want to be spammed or ruin my deliverability score with my SMTP provider.

Conclusion

In this post, I shared how to fetch external APIs in Next.js from a server component and how to prevent spam with a honey pot field. I hope this helps you in your Next.js journey. Stay tuned for more Next.js posts.

Deploying in AWS - Week 3/4: New NOTR Server

Aldo Portillo — Wed, 15 May 2024 23:26:49 +0000

Introduction

After two weeks of intensive learning and applying, I successfully deployed my server for Neat on the Rocks onto AWS. This journey involved learning and utilizing various AWS services, including ECS, EC2, ECR, IAM, RDS, and security groups. In this blog post, I'll walk you through the steps I took, the challenges I faced, and the solutions I found.

AWS Services Overview

Before diving into the steps, here's a brief overview of the AWS services I used:

ECS (Elastic Container Service): Manages and deploys Docker containers.
EC2 (Elastic Compute Cloud): Provides scalable computing capacity.
ECR (Elastic Container Registry): Stores and manages Docker container images.
IAM (Identity and Access Management): Manages access to AWS services and resources.
RDS (Relational Database Service): Manages relational databases in the cloud.
Security Groups: Controls inbound and outbound traffic to AWS resources.

Steps to Deploy the NOTR Server

Step 1: Setting Up the Docker Environment

First, I ensured that my application was containerized using Docker. I created a Dockerfile to define the environment and instructions for building the Docker image.

The code for this is available in my previous blog post where I containerized the application. The main difference is that I removed the creation for the Postgres image since we used AWS RDS.

Step 2: Pushing the Docker Image to ECR

When you open up AWS ECR, you can create a new container repository and get instructions on how to push. You need to have the Docker Daemon and AWS CLI installed.

Step 3: Setting Up ECS

I configured ECS to manage and deploy the Docker container. This involved creating a task definition and an ECS service.

The task definitions is where I defined my port routing and environment variables.

I created an ECS service to run and manage my tasks. This involved setting up a cluster, creating the service, and configuring the load balancer to route traffic to my tasks.

Step 4: Configuring Security Groups

I set up security groups to further secure my server to only allow certain IPs to access certain things. This was a difficult task since every AWS service has its own IP; I had to track them all.

Step 5: Setting Up RDS

I used Amazon RDS to manage my PostgreSQL database. This involved creating an RDS instance, setting up the database, and configuring my application to connect to it.

Step 6: IAM Roles and Policies
I created IAM roles and policies to manage access to AWS resources. This included roles for ECS tasks to access ECR, S3, and RDS.

Conclusion

Deploying Neat on the Rocks onto AWS was a challenging but rewarding experience. I gained valuable insights into using ECS, EC2, ECR, IAM, RDS, and security groups. By carefully configuring each component and troubleshooting issues as they arose, I successfully deployed my server and made it accessible on the internet.

User Authentication and Middleware - Week 2: New NOTR Server

Aldo Portillo — Wed, 01 May 2024 03:36:30 +0000

Welcome back to our journey through the server-side development of Neat on the Rocks (NOTR). This week was all about establishing a solid foundation for managing user interactions through MongoDB and setting up robust authentication mechanisms.

Connecting to MongoDB

Our first task was to integrate MongoDB, a database choice driven by its flexibility and capability to handle complex data structures efficiently. Using Mongoose, we streamlined the process of schema validation and business logic implementation.

Crafting the UserModel

const mongoose = require('mongoose');
const bcrypt = require('bcryptjs');

const User = new mongoose.Schema({
    email: { type: String, required: true, unique: true, select: false},
    password: { type: String, required: true, select: false},
    firstName: { type: String, required: true },
    lastName: { type: String, required: true },
    username: { type: String, required: true, unique: true },
    height: { type: Number, required: true },
    weight: { type: Number, required: true },
    dob: { type: Date, required: true },
    sex: { type: String, required: true },
    friends: [{ type: mongoose.Schema.Types.ObjectId, ref: 'User' }],
    friendRequests: [{ type: mongoose.Schema.Types.ObjectId, ref: 'User' }],
    photoUrl: { type: String, required: false },
    admin: { type: Boolean, required: false, default: false, immutable: true, select: false }
});

User.pre('save', async function(next) {
    if (!this.isModified('password')) return next();
    try {
        const salt = await bcrypt.genSalt(10);
        this.password = await bcrypt.hash(this.password, salt);
        next();
    } catch (error) {
        next(error);
    }
});


User.methods.comparePassword = async function(candidatePassword) {
    return await bcrypt.compare(candidatePassword, this.password);
};


const UserModel = mongoose.model('User', User);

module.exports = UserModel;

User Methods:

The UserModel includes methods such as comparePassword, which facilitates secure password verification during login. These methods enhance the security and functionality of our user management system.

Implementing Middleware for Security

Security is paramount, so we fortified our application with essential middleware:

Authentication Middleware (authJWT): Validates user sessions, ensuring that each request is authenticated.

Admin Field Stripping Middleware (stripAdminField): Prevents unauthorized manipulation of the admin field, safeguarding against potential vulnerabilities.

Authorization Middleware (verifyAdmin): Prevents any user that isn't an admin to access certain routes

Setting Up User Authentication

const loginUser = async (req, res) => {
    try {
        const { email, password } = req.body;
        const user = await UserModel.findOne({ email });
        if (!user || !(await user.comparePassword(password))) {
            return res.status(401).send('Authentication failed');
        }
        const token = generateAuthToken(user);
        res.send({ user, token });
    } catch (error) {
        res.status(500).send(error.message);
    }
}

Adding and Removing Friends

We added functionality to manage social interactions, such as sending friend requests and accepting or rejecting them. This enhances the social aspect of NOTR, allowing users to connect with each other within the application. Here is an example of a friend request.

const sendFriendRequest = async (req, res) => {
    const { userId, friendId } = req.body;
    const recipient = await UserModel.findById(friendId);
    if (!recipient) {
        return res.status(404).send('User not found');
    }

    if (recipient.friendRequests.includes(userId) || recipient.friends.includes(userId)) {
        return res.status(400).send('Request already sent or they are already friends');
    }
    recipient.friendRequests.push(userId);
    await recipient.save();
    res.send('Friend request sent');
}

Conclusion

This week was packed with creating the MVP for the social media aspect of the server. This feature is why I am recreating the server with more knowledge that I had 4 years ago. I am glad to get it working and look forward to testing my endpoints in the client.

Containerization and Data Storage - Week 1: New NOTR Server

Aldo Portillo — Wed, 24 Apr 2024 04:11:10 +0000

Introduction

Last week, I embarked on a journey to rebuild the Neat on the Rocks (NOTR) server using a MERN stack with an additional PostgreSQL database. This week, I've made substantial progress in containerizing the application, dealing with ORM issues, and implementing secure data storage solutions. In this post, I'll share the details of my journey and some key learnings that might help others tackling similar challenges.

Containerizing the Application

Containerization is a crucial step for ensuring that our application runs consistently across different environments. I started by creating a Dockerfile and a docker-compose.js file to manage the multi-container setup. Here's a brief look at what these files entail:

Dockerfile

# Use the official Node.js 16 as a base image
FROM node:16

# Set the working directory in the container
WORKDIR /usr/src/app

# Install netcat (nc), useful for checking port availability
RUN apt-get update && apt-get install -y netcat postgresql-client

# Copy package.json and package-lock.json
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy your application source code including the entrypoint script
COPY . .

# Ensure the entrypoint script is executable
RUN chmod +x ./entrypoint.sh

# Expose port 5000 for the application
EXPOSE 5000

# Use the entrypoint script to start the service
CMD ["./entrypoint.sh"]

Docker Compose

services:
  db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_DB: ${POSTGRES_DB}
  app:
    image: notr-server
    ports:
      - "8000:5000"
    depends_on:
      - db
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_HOST=db
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_PORT=${POSTGRES_PORT}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_REGION=${AWS_REGION}
      - AWS_S3_BUCKET_NAME=${AWS_S3_BUCKET_NAME}
    command: ["/usr/src/app/entrypoint.sh"]

The entry-point.sh script ensures that our database is ready before the web server starts, avoiding any startup issues.

I made tutorial on Github. Follow the README to containerize your first express application.

Transitioning from ORM to SQL Queries

Due to issues with the Sequelize ORM in a containerized environment, I decided to rewrite my models as classes with methods directly executing SQL queries. This approach provided more control and reduced unexpected behaviors.

Example Model Class


const pool = require('../db');

class Spirit {
    static async findAll() {
        const query = 'SELECT * FROM Spirits;';
        try {
            const result = await pool.query(query);
            return result.rows;
        } catch (err) {
            throw err;
        }
    }

    static async findById(id) {
        const query = 'SELECT * FROM Spirits WHERE id = $1;';
        try {
            const result = await pool.query(query, [id]);
            return result.rows[0];
        } catch (err) {
            throw err;
        }
    }

    static async create(data) {
        const { id, name, abv, sugarConcentration, acid, calories, ethanol, fat, carb, sugar, addedSugar, protein, spiritCategoryId } = data;
        const query = `INSERT INTO Spirits (id, name, abv, sugarConcentration, acid, calories, ethanol, fat, carb, sugar, addedSugar, protein, spirit_category_id)
                       VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
                       RETURNING *;`;
        const values = [id, name, abv, sugarConcentration, acid, calories, ethanol, fat, carb, sugar, addedSugar, protein, spiritCategoryId];
        try {
            const result = await pool.query(query, values);
            return result.rows[0];
        } catch (err) {
            throw err;
        }
    }

}

module.exports = Spirit;

Implementing AWS S3 for Secure Data Storage

Storing structured data securely is paramount. I implemented AWS S3 buckets to store some of the structured data from MongoDB before moving to PostgreSQL. Here's how I integrated S3 with my Node.js application:

const { S3Client, GetObjectCommand } = require('@aws-sdk/client-s3');
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");
const csv = require('csv-parser');
const { finished } = require('stream/promises');
const dotenv = require('dotenv').config();

// AWS SDK Configuration
const s3Client = new S3Client({
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
  region: process.env.AWS_REGION
});

async function downloadAndParseCSV(fileName) {
  const params = {
    Bucket: process.env.AWS_S3_BUCKET_NAME,
    Key: fileName
  };

  try {
    const command = new GetObjectCommand(params);
    const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });
    const fetch = (await import('node-fetch')).default; // Doing this since node-fetch is an ESM module
    const response = await fetch(url);
    const stream = response.body.pipe(csv());
    const data = [];
    stream.on('data', (row) => data.push(row));
    await finished(stream);
    return data;
  } catch (err) {
    console.error(`Error downloading or parsing ${fileName} from S3:`, err);
    throw err;
  }
}

module.exports = { downloadAndParseCSV };

Conclusion

This week was filled with challenges and learning opportunities as I continued to rebuild the NOTR server. The move to containerization, direct SQL management, and implementing AWS S3 has set a solid foundation for future development. Stay tuned for more updates as I begin implementing Users backed by MongoDB!

Migrating Data from MongoDB to PostgreSQL - Week 0: New NOTR Server

Aldo Portillo — Mon, 15 Apr 2024 18:19:37 +0000

Introduction

In week 0 of modifying my server for Neat on the Rocks, I focused on transferring part of my data from MongoDB (a non-relational database) to PostgreSQL (a relational database). This migration is critical as I aim to use SQL's robust querying capabilities for complex searches in my cocktail database, while maintaining a non-relational database for the social media features of the app.

Creating Models

I started by setting up Sequelize models for PostgreSQL. This was essential to ensure data integrity and establish relationships that are crucial for the relational database part of my stack. Here’s how I defined the Cocktail model:


const { Model, DataTypes } = require('sequelize');

module.exports = (sequelize, DataTypes) => {
    class Cocktail extends Model {}

    Cocktail.init({
        name: DataTypes.STRING,
        description: DataTypes.STRING,
        glasswareId: DataTypes.INTEGER,
        imageUrl: DataTypes.STRING,
        techniqueId: DataTypes.INTEGER,
        garnishId: DataTypes.INTEGER
    }, {
        sequelize,
        modelName: 'Cocktail'
    });

    Cocktail.associate = models => {
        Cocktail.belongsTo(models.Glassware, { foreignKey: 'glasswareId' });
        Cocktail.belongsTo(models.Technique, { foreignKey: 'techniqueId' });
        Cocktail.belongsTo(models.Garnish, { foreignKey: 'garnishId' });
    };

    return Cocktail;
};

Aggregation Pipeline

Due to the unstructured format of my MongoDB data, I had to prepare it for a relational database, so I used a MongoDB aggregation pipeline to create rows for my data.

db.cocktails.aggregate([
  { $unwind: "$specs" },
  { $project: {
    id: "$_id",
    cocktailName: "$name",
    spirit: "$specs.spirit",
    ounces: {
      $cond: {
        if: { $ifNull: ["$specs.ounces", false] },
        then: "$specs.ounces",
        else: "N/A"
      }
    }
  }}
]);

This allowed me to take multiple cocktails structured like this:

{
  "_id": {
    "$oid": "1"
  },
  "name": "Old Fashioned",
  "image": {
    "filePath": "https://images.unsplash.com/photo-1470337458703-46ad1756a187?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1169&q=80"
  },
  "specs": [
    {
      "spirit": "Bourbon (80 proof)",
      "ounces": 2
    },
    {
      "spirit": "Simple syrup (1:1)",
      "ounces": 0.5
    },
    {
      "spirit": "Angostura bitters",
      "dashes": 2
    }
  ],
  "instructions": [
    "Fill the mixing glass with ice cubes.",
    "Pour in the cocktail ingredients.",
    "Stir for 30 seconds",
    "Use a strainer to pour into a glass.",
    "Add garnishes and Enjoy!"
  ],
  "description": "Indulge in the distinguished charm of an Old Fashioned cocktail. Crafted with the utmost reverence for tradition, this timeless concoction marries the deep notes of whiskey with the delicate sweetness of muddled sugar. Aromatic bitters add a touch of complexity, while a citrus twist imparts a subtle brightness. Served over a single large ice cube, the Old Fashioned beckons with sophistication. Garnished with an orange twist or a maraschino cherry, it stands as an ode to the classic art of mixology.",
  "method": "Stirred",
  "garnish": "Orange Peel",
  "type": "cocktail",
  "glass": "Rocks",
  "__v": 0
}

And extract the specs of each cocktail so that each index of the spec array of every cocktail becomes a column in my specs table.

Conclusion

After learning system design, this is something I have been wanting to implement for a while, but I was hesitant to transfer from a non-relational DB to a relational DB in Node. It wasn't too bad. I will still leverage the perks of a non-relational DB in the project for the social media aspect.

Authorization

Aldo Portillo — Wed, 15 Nov 2023 02:11:38 +0000

Authorization

Introduction

Although I am fortunate enough to develop software during the week, I still get the pleasure of being employed as a server and bartender. My current service industry employer is making huge changes on structure, and I am very excited for it; however, with change there is always resistance. During one of our wine meetings, there was talk about having an anonymous box where employees can submit anonymously make comments about the changes and suggestions. Considering we have 4 locations, and I was speaking to one of the higher ups. I thought it would be a great idea to create a anonymous box backend.

Models

In terms of structuring the database, Users and Posts were a given. At first, I thought to myself since they were anonymous. I can just render all the posts to the admin and none to the user; however, I had to think ahead a bit. Although the suggestion was to only use it for the Oak Park location, I am hopeful that I will be used for our other locations. So I also made a restaurants table, to allow admin to view posts by location.

So what did I need to do to secure the posts from the outside world? Easy, authenticate user. If user isn't authenticated, make them login or register. It's miller time. No its not. We don't really care if some random person in the world reads what Bob had to say about his coworker Mallory (kinda), we care about making sure Mallory never reads what Bob said. Well lets find those vulnerabilities.

Routes, Controllers and Actions

There are a some routes that we want some users to access but not others:

Post Routes: We want the user to either be admin or author view the post; otherwise, redirect to home. Although there is no direct link in the app to navigate to one's post. It doesn't take a rocket scientist to figure out the routes to a RESTful app.

There are a some routes that we want some users to access but not others:

Restaurant Routes: We only want the admin to view these routes. Since not only do they contain the power to create restaurants and read all posts associated with the restaurant, but they also can delete a restaurant. That is a huge loss of data. Not good.

Implementation

Include Pundit in ApplicationController

#app/controllers/application_controller.rb
include Pundit

Generate Pundit Policy Files
Run "rails g pundit:policy policy_name" in bash

Write authorization logic in policy files. For posts, I actually used a scope.

# In app/policies/post_policy.rb
class PostPolicy < ApplicationPolicy
    attr_reader :user, :post

    def initialize(user, post)
        @user = user
        @post = post
    end

    def index?
        user.present?
    end

    def show?
        scope.where(:id => post.id).exists?
    end

    def update?
        user.admin? || post.user == user
    end

    def destroy?
        user.admin? || post.user == user
    end

    class Scope < Scope
        def resolve
            user.admin? ? scope.all : scope.where(user: user)
        end
    end
end

Implement Authorization in Controllers

#In app/controllers/post_controller.rb
class PostController < ApplicationController
    ...
    # GET /posts or /posts.json
    def index
        # @posts = Post.all
        @posts = policy_scope(Post)
    end
    def show
        authorize @post
    end
    ...
end

Handle Pundit::NotAuthorizedError in Application Controller by rescuing it.

#In app/controllers/application_controller.rb
rescue_from Pundit::NotAuthorizedError, with: :user_not_authorized

private

def user_not_authorized
flash[:alert] = "You are not authorized to perform this action."
redirect_to(request.referrer || root_path)
end

Conclusion

And just like that, the posts are protected from other users and the world. The only people allowed to see them are admin and author. Also I definitely recommend Vezzi Barbera. Had 2 glasses when writing this article. Nice medium body+, slightly acidic, dry wine.

CAP Theorem

Aldo Portillo — Sat, 11 Nov 2023 14:52:46 +0000

Theorem: Distributed systems can only guarantee 2 out of 3: CAP.

Acronym

C - Consistency - If two requests are made from different sources against different "nodes" off the distributed store, you will get the same data returned.
A - Availability - If two requests are made from different sources against different "nodes" off the distributed store, you will get a response.
P - Partition Tolerance - If nodes are disconnected, the system will continue to work. Assumed as constant in distributed systems.

DB Types

CA Database - Consistent and Available

Not partition tolerant.

Databases - SQL: PostgreSQL and MySQL.

AP Database - Available and Partition Tolerant

Access to the same dataset but no guarantee every request will receive the same response.

Databases - NoSQL: Mongo and HBase

CP Database - Consistent and Partition Tolerant

Guaranteed that every request receives the same response.

Databases: Spanner, Dynamo, Cosmos and Cockroach

Implementing Consistency

Foundations: Raft and MVCC

Raft

Distributed Consensus Algorithm provides atomic writes and consistent reads.

Raft Leader

Leader is elected
Coordinates all writes, proposes commands to followers
Only allowed to serve authoritative up-to-date

Atomic Writes(Replication)

Commands are proposed to the Raft Leader replica and distributed to followers but only accepted when a quorum of followers have acknowledged receiving it. This helps with consistency.

Multiversion concurrency control - MVCC

Ensures consistent data is always present without any overlap for transactions.

Applying CAP in real world for comprehension

Bank

Two ATMs support 3 operations: withdraw, deposit and check balance. These ATMs don't have a central database.

Prioritizing Consistency - The ATM may refuse to accept deposits or withdraws when there is a partition. This leads to a sad customer.

Prioritizing Availability - The customer can withdraw the full balance from both ATMs when there is a partition. This leads to a negative balance.

Conclusion

This is just a high level overview on the CAP theorem. The world is more complex than this. CAP theorem assumes there is a partition, but if there isn't, we also have latency vs consistency in the PACELC theorem.

Docker

Aldo Portillo — Sat, 11 Nov 2023 04:50:58 +0000

Introduction

Before VMs, every server only had one operating system. This means if you wanted a windows server and linux server. You needed two physical servers. Here comes virtualization, instead of installing an OS you install a hypervisor (VMware ESXi). This allows you to divide your server resources into multiple servers running different OS.

Hypervisor Flow

<Mermaid name="diagram">
flowchart TD
    A[Hardware] --> B(Hypervisor)
    B --> D{Windows Server}
    B --> E{Ubuntu Server}
    B --> F{Debian Server}
</Mermaid>

Well why use Docker to run different OS on your server if hypervisors do that already? Well, virtualization and virtual machines virtualize hardware. Docker visualizes the operating system.

Docker Flow

flowchart TD
    A[Hardware] --> C{Ubuntu Server}
    C --> D(Docker Engine)
    D --> E{Ubuntu Server}
    D --> F{Debian Server}
    D --> G{Centos Server}

The docker containers contain the OS that we need to run. These are micro containers that contain their own OS, CPU, Memory and Network. These are lightweight, fast and isolated.

Why is it so fast?

Since we are building our docker engine on top of our linux server. We only need one kernel. All Docker containers use the same kernel. As opposed to, hypervisors which contain their own kernels.

How is the industry using Docker?

Developers can write their code. Deploy it in a docker container, and it works anywhere.

Microservices

Take portions of your stack and segmenting it into smaller bits and pieces. Split your app server, app client and database in separate containers.

Dockers in Node

Dockerfile is a blueprint for building a Docker Image

A Docker Image is a template for running Docker Containers.

A Container is just a running process.

Developer creating the software defines the environment with the dockerfile.
Another developer can use the docker file to rebuild the environment which is saved as an image. Images can be shared and anyone can pull that image.
The image can be run in a docker container

Creating a Dockerfile that Creates Docker Image

//In src/index.js

const app = require ("express")() ;
app.get ('/', (req, res ) =>
    res.json({ message: 'Docker is easy!' })
);
const port = process.env.PORT || 8080;

app.listen(port, () => console. log(`app listening on http://localhost:${port}`))

Create Docker Ignore and add node_modules

#In Dockerfile

# From Node Image
FROM node:12

# Set directory
WORKDIR /app

#Install dependencies first so they can be cached

COPY package*.json ./

RUN npm install

# Copy source code 
COPY . .

# Set Port
ENV PORT=8080

EXPOSE 8080

#Run
CMD["npm", "start"]

Create Docker Image

docker build -t username/app:1.0 .

The period is the path. When the process is done. It returns "successfully built image_id"

Next you can push this to the cloud for others to use. For now we are gonna, run the docker locally.

Create container

docker run image_id

docker run -p 5000:8080

Volume: share data between containers

#Create volume
docker volume create shared-data-folder
#mount volume
docker run \
> --mount source=shared-data-folder,target=/data-folder

Docker Compose: In Progress

Command Cheat Sheet


#create new docker container w
docker run -d -t --name container_name centos/ubuntu/node

#show all containers in server
docker ps

#show resources for all containers
docker stats

Live Reloading in Rails

Aldo Portillo — Wed, 01 Nov 2023 16:59:30 +0000

Introduction

When developing the frontend of any application, the most tedious thing is refreshing the client/server. Luckily, when it came to HTML, I had live server; and when it came to Node, I had Nodemon. When it came to ruby, the server refreshed but not the client.

Rails Live Reload

Is a Rails live reloader for views, partials, css, and js. This means that every single time you save a .erb, .css or .js file, the page will reload to show the changes.

Configuration

In Gemfile add:

group :development do
  gem "rails_live_reload"
end

In order to only download this gem in development. Then run bundle.

Create Initializer

#Create config/initializers/rails_live_reload.rb and add
RailsLiveReload.configure do |config|
  # config.url = "/rails/live/reload"

  # Default watched folders & files
  # config.watch %r{app/views/.+\.(erb|haml|slim)$}
  # config.watch %r{(app|vendor)/(assets|javascript)/\w+/(.+\.(css|js|html|png|jpg|ts|jsx)).*}, reload: :always

  # More examples:
  # config.watch %r{app/helpers/.+\.rb}, reload: :always
  # config.watch %r{config/locales/.+\.yml}, reload: :always

  # config.enabled = Rails.env.development?
end if defined?(RailsLiveReload)

Enjoy! With just two steps you saved yourself hours of refreshing the client in Rails!

VSCode Extensions

Aldo Portillo — Sat, 28 Oct 2023 20:13:02 +0000

Introduction

As I switch between dev environments. I find myself missing my extensions. Here is a list of some of my extensions, so I can refer back to them or, so you can can also use them! Comment your favorite extensions!

Global Workspace

Name	Description
Error Lens	Improve highlighting of errors, warnings and other language diagnostics.
VSCode Icons	Icons for Visual Studio Code
Output Colorizer	Syntax highlighting for log files
DotENV	Syntax highlighting for your .env file
Code Spell Check	Spell checker
Rainbow CSV	Highlight CSV and TSV files, Run SQL-like queries
Markdown All in One	All you need to write Markdown (keyboard shortcuts, table of contents, auto preview and more)
Markdown Lint	Markdown linting and style checking for Visual Studio Code
CodeSnap	Take beautiful screenshots of your code
Docker	Build, manage, and deploy containerized applications.
WakaTime	Track the time you spent coding
Indent Rainbow	Make indentation easier to read
Blockman	Mark and highlight code blocks

HTML and CSS

Name	Description
Auto Rename Tag	Auto rename paired HTML/XML tag

Ruby

Name	Description
Ruby LSP	VS Code plugin for connecting with the Ruby LSP
Endwise	Wisely add "end" in Ruby.
ERB Formatter	Format/Beautify ERB files
Rails DB Schema	Definition and Completion provider for Rails DB Schema.
Rails Routes Navigator	Display Routes of Rails, search incremental and navigate to the source location
Ruby and Rails Snippets	Snippets for ruby and rails
Rubocop	Rubocop gives suggestions on how to write Ruby code in a proper way.

Non Essential

Name	Description
VSCode Pets	Pets for your VS Code
In your face	In Your Face, shows you Incredible 'Ouch Faces' that correlate to the number of errors in your code!
Subway Surfers	Subway Surfers opens a window in vscode with game play footage to stimulate your short attention span

Web Scraping

Aldo Portillo — Fri, 27 Oct 2023 20:15:31 +0000

I am currently working on a Ruby on Rails project that allows MMA promoters to create events and MMA fighters to find matches. In order to see the entire workflow of the app, I need some data; however, I wanted the data to be similar to the data I will require for the user. Fortunately, the UFC exists and their website has images, names, reach, weights, and heights. Unfortunately, the UFC does not offer an API for this data. So I had to web scrape it. Below is the code I wrote to scrape the fighters data using a csv of their slugs.

Tools

Steps

Iterate through the slugs in the CSV in lib/sample_data/athletes_slugs.csv
Make a http request to "https://www.ufc.com/athlete/#{slug}"
Parse the request using Nokogiri to get the document
Scrape data you need using classes
Save them in CSV if they don't exist

Code

#In lib/tasks/scrape.rake

desc "Scrape Athlete Metric Data"
task({ :scrape_athlete_metrics => :environment }) do

  #STEP 1: Get the slugs from CSV
  slugs = CSV.read('lib/sample_data/athletes.csv').map { |row| row.at(1) }

  #STEP 2: Make a request to the URL with each slug
  raw_responses = slugs.map { |slug| HTTP.get("https://www.ufc.com/athlete/#{slug}") }

  #STEP 3: Parse the request using Nokogiri
  documents = raw_responses.map { |response| Nokogiri::HTML(response.to_s) }

  #STEP 4: Scrape Data
  fighters_data = documents.map do |doc|
    image_element = doc.at('img.hero-profile__image')
    image_src = image_element ? image_element.attr('src') : nil

    {
      :name => doc.css('.hero-profile__name').text.strip,
      :image_src => image_src,
      :age => doc.at('div:contains("Age") > .c-bio__text')&.text&.strip,
      :reach => doc.at('div:contains("Reach") > .c-bio__text')&.text&.strip,
      :height => doc.at('div:contains("Height") > .c-bio__text')&.text&.strip,
      :weight => doc.at('div:contains("Weight") > .c-bio__text')&.text&.strip,
    }
  end

  # STEP 5: Save in CSV
  CSV.open('lib/sample_data/athletes_metrics.csv', 'a+') do |csv|
    existing = csv.entries
    fighters_data.each do |fighter|
      unless existing.include?([fighter[:name], fighter[:image_src], fighter[:age], fighter[:reach], fighter[:height], fighter[:weight]])
        csv << [fighter[:name], fighter[:image_src], fighter[:age], fighter[:reach], fighter[:height], fighter[:weight]]
      end
    end
  end

Roadblocks

I faced two roadblocks when scraping data. The first one is something I noticed when writing the code above which I was able to work around. The other roadblock there is no workaround using Nokogiri.

Non specific class names:
When trying to get the height, weight, reach and age of the fighters, I noticed they had non descriptive class names. They all had the same class name:
```
<div class="c-bio__field">
    <div class="c-bio__label">Height</div>
    <div class="c-bio__text">65.00</div>
</div>
```
Luckily since the div that displays the text height is in the same div as the div that displays the integer. We can select them as:
```
:height => doc.at('div:contains("Height") > .c-bio__text')&.text&.strip,
```
This selects the parent div that contains "height" and gets the child div containing the integer.
Javascript rendering html:
This is an issue I was unable to resolve directly. I was trying to get the names and locations of venues across america by web scraping since there was no free API. I found a great page with all the data I needed, so I wrote the best web scraper the world has ever seen and it failed. I opened the chrome debugger and realized the elements weren't initially loaded. Nokogiri only grabs the html before any javascript is ran.

After many different approaches, I decided to throw in the towel. Well at this point I should just delete the repository and start a different project. Wrong. Thank god for google dorking.

I had to understand that all the data I was getting from the scraper was stored in a csv. Well what if a csv with the data already exists somewhere on the world wide web? Time to google dork.

I searched venues filetype:pdf on google and after some searching I found a CSV containing latitude, longitude, address and name. Exactly what I was hoping to web scrape.