DEV Community: Javier Andre Neira Machaca

Building an App with a Cloud NoSQL Server Database

Javier Andre Neira Machaca — Wed, 18 Dec 2024 05:16:02 +0000

Introduction

The proliferation of cloud technologies has made it easier than ever to build scalable, efficient, and reliable applications. Choosing a database is one of the most critical decisions in app development, especially when working with modern applications that require high scalability and flexibility. In this article, we will explore how to build an application with a cloud NoSQL database, excluding DynamoDB and CosmosDB. We'll use Couchbase, a powerful NoSQL database, as an example.

Why Choose a NoSQL Database?

NoSQL databases are designed to handle unstructured or semi-structured data efficiently. They are ideal for applications with high scalability requirements, such as social networks, IoT platforms, and real-time systems. Key advantages of NoSQL databases include:

Scalability: Horizontal scaling ensures that the database can handle increased workloads.
Flexibility: Schema-less design allows for rapid iteration during development.
Real-time Processing: Many NoSQL databases support real-time data synchronization. ## What is Couchbase? Couchbase is a NoSQL document database designed for high-performance applications. It offers a unique combination of scalability, flexibility, and powerful querying capabilities, making it ideal for modern web and mobile applications.

Key Features}

Distributed Architecture: Ensures high availability and scalability.
N1QL Query Language: SQL-like query language for JSON data.
Real-time Sync: Synchronize data across devices with Couchbase Mobile.
Flexible Data Model: Store data in JSON format.
Offline Support: Built-in support for offline-first applications.

Setting Up Couchbase for Your App

Step 1: Install Couchbase Server

Download Couchbase Server from the official website.
Follow the installation instructions for your operating system.
Start the Couchbase Server and access the web console at http://localhost:8091.

Step 2: Create a Bucket

Log in to the Couchbase web console.
Navigate to the Buckets section and create a new bucket for your application data.

Step 3: Set Up Couchbase SDK

Install the Couchbase SDK for your platform. For example, in a Node.js application:

npm install couchbase

Initialize Couchbase in your app:

const couchbase = require('couchbase');

const cluster = new couchbase.Cluster('couchbase://127.0.0.1', {
  username: 'Administrator',
  password: 'password',
});
const bucket = cluster.bucket('your-bucket-name');
const collection = bucket.defaultCollection();

Step 4: Define Your Data Model

Couchbase uses a document model to store data. Here’s an example structure for a task management app:

Tasks (Bucket)

Document 1

id: "1"
title: "Build Couchbase App"
description: "Learn how to integrate Couchbase."
status: "In Progress"
Document 2
id: "2"
title: "Publish Article"
description: "Publish on Dev.to."
status: "Pending"

Step 5: CRUD Operations

Add Data

const addTask = async (task) => {
  try {
    await collection.insert(task.id, task);
    console.log("Task added successfully.");
  } catch (err) {
    console.error("Error adding task: ", err);
  }
};

Read Data

const fetchTasks = async () => {
  try {
    const query = "SELECT * FROM `your-bucket-name`";
    const result = await cluster.query(query);
    result.rows.forEach((row) => {
      console.log(row);
    });
  } catch (err) {
    console.error("Error fetching tasks: ", err);
  }
};

Update Data

const updateTask = async (taskId, updatedFields) => {
  try {
    await collection.mutateIn(taskId, [
      couchbase.MutateInSpec.upsert("status", updatedFields.status),
    ]);
    console.log("Task updated successfully.");
  } catch (err) {
    console.error("Error updating task: ", err);
  }
};

Delete Data

const deleteTask = async (taskId) => {
  try {
    await collection.remove(taskId);
    console.log("Task deleted successfully.");
  } catch (err) {
    console.error("Error deleting task: ", err);
  }
};

Deployment

Once your application is ready, deploy it to your chosen platform. Couchbase integrates well with cloud platforms like AWS, Azure, and Google Cloud. You can also use Docker to containerize your application:

docker run -d --name couchbase -p 8091-8096:8091-8096 -p 11210:11210 couchbase

Conclusion

Using a cloud NoSQL database like Couchbase can streamline app development by providing high performance, scalability, and ease of use. By following the steps outlined in this article, you can build robust applications that leverage the power of modern NoSQL databases.

Construyendo una aplicación con Change Data Capture (CDC) utilizando Debezium, Kafka y NiFi

Javier Andre Neira Machaca — Sat, 14 Dec 2024 17:08:18 +0000

En este artículo aprenderás a crear una aplicación con capacidades de Change Data Capture (CDC) utilizando Debezium, Apache Kafka y Apache NiFi. Este enfoque te permitirá capturar, procesar y transmitir cambios en tus datos en tiempo real, habilitando flujos de datos modernos y reactivos. Al final del artículo encontrarás un enlace a un repositorio de GitHub con código de ejemplo para que puedas replicar el entorno en tu máquina.

¿Qué es el Change Data Capture (CDC)?

El Change Data Capture (CDC) es un patrón que permite capturar las mutaciones de datos (inserciones, actualizaciones y eliminaciones) ocurridas en un sistema de origen. En lugar de depender de lecturas periódicas (batch) que pueden ser costosas e ineficientes, el CDC detecta y emite eventos tan pronto como se producen, lo que facilita la replicación, sincronización y análisis en tiempo real.

Algunas ventajas del CDC:

Sincronización de datos en tiempo real: Mantener múltiples sistemas alineados sin importar la frecuencia de cambios.
Análisis en vivo: Procesar datos en el momento exacto en que ocurren los eventos, ideal para dashboards, alertas o análisis inmediato.
Menor latencia: Comparado con procesos ETL tradicionales, se evitan los “batch windows”.
Escalabilidad y resiliencia: Integrado con sistemas de mensajería distribuida como Kafka, se logra un pipeline flexible y escalable.

Herramientas Clave en la Arquitectura

Debezium: Una plataforma de CDC que se integra con distintos motores de base de datos (MySQL, PostgreSQL, Oracle, MongoDB, etc.). Debezium captura los cambios a nivel de binlog (o equivalente) y los convierte en eventos, que luego publica en Kafka.
Apache Kafka: Un sistema de mensajería distribuido, escalable y altamente disponible. Sirve como capa intermedia para almacenar y transmitir los eventos generados por Debezium. Los consumidores pueden leer estos eventos en tiempo real.
Apache NiFi: Una herramienta de integración y automatización de flujos de datos basada en una interfaz gráfica (low-code), que facilita la transformación, enrutamiento y distribución de datos hacia sistemas destino (bases de datos analíticas, data lakes, almacenamiento en la nube, etc.).

Arquitectura General del Pipeline

La arquitectura típica consta de:

Base de Datos Origen: Donde se ejecutan transacciones. Por ejemplo, una base MySQL.
Debezium: Detecta cambios y produce eventos.
Kafka: Recibe eventos desde Debezium y los pone a disposición de cualquier consumidor.
NiFi u otras Aplicaciones: Consumen, transforman y reenvían estos datos a sistemas de destino (por ejemplo, ElasticSearch, HDFS, otra base de datos).
Servicios Destino: Donde se utilizan los datos (reportes, dashboards, aplicaciones web, análisis en tiempo real).

Pasos para Implementar el Pipeline

1. Configurar la Base de Datos de Origen

Asume que usas MySQL. Asegúrate de habilitar el binlog y el formato de registro adecuado (ROW):

[mysqld]
log_bin = mysql-bin
binlog_format = ROW
server_id = 1

Crea un usuario con permisos para que Debezium pueda leer el binlog.

2. Levantar Kafka y Debezium con Docker

Crea un archivo docker-compose.yml similar a:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.0.1
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - "9092:9092"

  debezium:
    image: debezium/connect:1.9
    ports:
      - "8083:8083"
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_config
      OFFSET_STORAGE_TOPIC: debezium_offset
      STATUS_STORAGE_TOPIC: debezium_status
    depends_on:
      - kafka

3. Configurar el Conector Debezium

Con Debezium en marcha, registra un conector que apunte a tu base de datos MySQL. Por ejemplo:

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
  localhost:8083/connectors/ -d '{
    "name": "mysql-connector",
    "config": {
      "connector.class": "io.debezium.connector.mysql.MySqlConnector",
      "tasks.max": "1",
      "database.hostname": "tu_mysql_host",
      "database.port": "3306",
      "database.user": "debezium",
      "database.password": "debeziumpwd",
      "database.server.id": "184054",
      "database.server.name": "fullfillment_app",
      "database.include.list": "mydb",
      "database.history.kafka.bootstrap.servers": "kafka:9092",
      "database.history.kafka.topic": "schema-changes.mydb"
    }
  }'

4. Consumir Eventos con Apache NiFi

En NiFi, utiliza un ConsumeKafkaRecord_2_x para leer eventos de un tópico específico. Luego, con procesadores como JoltTransformJSON puedes transformar el payload. Finalmente, envía los datos con PutDatabaseRecord a otra base de datos, o PutElasticsearchHttpRecord a un índice Elasticsearch.

El flujo podría ser:
[ConsumeKafkaRecord_2_x] -> [JoltTransformJSON] -> [PutDatabaseRecord]

Adapta los procesadores según tus necesidades y ajusta las propiedades (tópico Kafka, host de destino, etc.).

Creating a To-Do List App with Firebase Firestore

Javier Andre Neira Machaca — Thu, 14 Nov 2024 05:18:27 +0000

In this post, I will show you how to create a to-do list application using HTML, CSS, JavaScript, and Firebase Firestore. This guide will help you understand how to integrate a cloud database with a simple web application, providing a complete experience from setup to implementation.

Technologies We Will Use:

HTML/CSS: For the structure and design of the application, ensuring that the interface is intuitive and easy to use.
JavaScript: For the application logic, adding interactivity and communicating with Firebase. This includes manipulating the DOM, handling events, and performing CRUD (Create, Read, Update, Delete) operations.
Firebase Firestore: As our cloud database to store tasks. Firestore allows us to store structured data and access it in real time.

Project Structure
Our project has the following file structure:
firebase-todo-app/ | ├── src/ │ ├── css/ │ │ └── styles.css │ └── js/ │ └── app.js │ ├── index.html └── README.md

index.html is the main file that loads in the browser and contains the base structure of the application.
styles.css contains the styles that make our application look good and visually appealing.
app.js is where all the JavaScript logic for handling tasks resides, including integration with Firebase Firestore.

Step 1: Firebase Setup
Before starting with the code, you need to configure Firebase. You can create a new project from Firebase Console, enable Firestore Database, and copy the project configuration.

Then, in the app.js file, add the Firebase configuration:

const firebaseConfig = {
    apiKey: "YOUR_API_KEY",
    authDomain: "YOUR_AUTH_DOMAIN",
    projectId: "YOUR_PROJECT_ID",
    storageBucket: "YOUR_STORAGE_BUCKET",
    messagingSenderId: "YOUR_MESSAGING_SENDER_ID",
    appId: "YOUR_APP_ID"
};

firebase.initializeApp(firebaseConfig);
const db = firebase.firestore();

With this, we have connected our application to Firebase Firestore.

Step 2: Creating the User Interface
In our index.html file, we have a simple structure that includes:

A title (<h1>) for the application.
A dropdown () to filter tasks (all, completed, pending).
A container () to display our tasks.
An input field () and a button () to add new tasks.

A container for messages () where we will display status messages for the user.

Here is the HTML code:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Firebase To-Do App</title>
    <link rel="stylesheet" href="css/styles.css">
</head>
<body>
    <h1>Firebase To-Do List</h1>
    <select id="filter">
        <option value="all">All</option>
        <option value="completed">Completed</option>
        <option value="pending">Pending</option>
    </select>
    <div id="todo-container">
        <p>Loading tasks...</p> <!-- Initial loading message -->
    </div>
    <input type="text" id="todo-input" placeholder="New task...">
    <button id="add-button">Add Task</button>
    <!-- Container for messages -->
    <div id="message"></div>

    <!-- Load Firebase from CDN (compat version) -->
    <script src="https://www.gstatic.com/firebasejs/9.6.1/firebase-app-compat.js"></script>
    <script src="https://www.gstatic.com/firebasejs/9.6.1/firebase-firestore-compat.js"></script>

    <!-- Your JavaScript file -->
    <script src="js/app.js"></script>
</body>
</html>

Step 3: Styling with CSS
The styles.css file allows us to give a pleasant visual style to the application. Here are some of the styles we applied:

General Design: We use a sans-serif font to give the application a modern and clean look.

Task Container: The container (#todo-container) has rounded borders, shadow, and a white background to stand out against the general background. This helps keep tasks organized and well defined.
Buttons: The "Add Task" button has a green background that changes when hovered over to indicate that it is active and to invite the user to interact with it.

Step 4: JavaScript Logic
In the app.js file, we define the logic for adding, displaying, editing, and deleting tasks. Firebase allows us to store tasks in the database and listen for changes in real time.

Adding a Task

To add a new task, we use the addTodo() function that takes the value from the input field and saves it in Firestore. Additionally, we ask the user if they want to add an optional due date:

async function addTodo() {
    const todoInput = document.getElementById("todo-input").value;
    const dueDateInput = prompt("Due date (optional): YYYY-MM-DD");
    const messageDiv = document.getElementById("message");

    if (todoInput) {
        try {
            const createdAt = new Date();
            const dueDate = dueDateInput ? new Date(dueDateInput) : null;

            await db.collection("todos").add({ text: todoInput, completed: false, createdAt, dueDate });
            document.getElementById("todo-input").value = "";
            showMessage("Task added successfully!", "green");
        } catch (error) {
            showMessage("Error adding task: " + error.message, "red");
        }
    } else {
        showMessage("Please enter a task.", "red");
    }
}

In this code:

We validate that the input field is not empty.
We create an object with the creation date and optionally a due date.
We use db.collection("todos").add() to save the task in Firestore.
We display a success or error message using the showMessage() function.
Displaying Tasks in Real Time

The loadTodos() function listens for changes in the database using onSnapshot() and displays all tasks in the interface. This function also allows filtering tasks by their status:

function loadTodos(filter = "all") {
    let query = db.collection("todos");

    if (filter === "completed") {
        query = query.where("completed", "==", true);
    } else if (filter === "pending") {
        query = query.where("completed", "==", false);
    }

    query.onSnapshot((snapshot) => {
        const todoContainer = document.getElementById("todo-container");
        todoContainer.innerHTML = "";
        snapshot.forEach((doc) => {
            const taskData = doc.data();
            const taskId = doc.id;

            const task = document.createElement("div");
            task.textContent = taskData.text;

            // Display creation and due dates
            if (taskData.createdAt) {
                const createdDate = new Date(taskData.createdAt.seconds * 1000);
                task.textContent += ` (Created: ${createdDate.toLocaleDateString()})`;
            }

            if (taskData.dueDate) {
                const dueDate = new Date(taskData.dueDate.seconds * 1000);
                task.textContent += ` (Due: ${dueDate.toLocaleDateString()})`;
            }

            if (taskData.completed) {
                task.style.textDecoration = "line-through";
                task.style.color = "gray";
            }

            const completeButton = document.createElement("button");
            completeButton.textContent = "✔️";
            completeButton.style.marginLeft = "10px";
            completeButton.onclick = () => markAsCompleted(taskId);

            const editButton = document.createElement("button");
            editButton.textContent = "✏️";
            editButton.style.marginLeft = "5px";
            editButton.onclick = () => editTask(taskId, taskData.text);

            const deleteButton = document.createElement("button");
            deleteButton.textContent = "🗑️";
            deleteButton.style.marginLeft = "5px";
            deleteButton.onclick = () => deleteTask(taskId);

            task.appendChild(completeButton);
            task.appendChild(editButton);
            task.appendChild(deleteButton);
            todoContainer.appendChild(task);
        });
    });
}

In this code:

We filter tasks by status (completed, pending, or all).
We listen for changes in real time with onSnapshot() to keep the interface automatically updated.
We dynamically create HTML elements to display each task, along with buttons to complete, edit, and delete.

Final Thoughts
Creating a to-do list application with Firebase is an excellent way to learn how to work with cloud databases and understand how to build a complete application with HTML, CSS, and JavaScript. Throughout this project, we explored how to add, edit, delete, and display tasks using Firestore, which provides real-time synchronization so users always have updated data.
Firebase makes it easy to implement many of the features needed for modern applications, such as cloud data persistence and real-time updates. With the right structure and development best practices, this project can be extended to add more advanced features like user authentication, notifications, or file storage.
I hope this guide helps you learn and develop your own applications with Firebase. If you have any questions or suggestions, feel free to leave a comment!
https://github.com/Dray-Lexter/firebase-todo-app.git