DEV Community: Ronny Medina

Recomendaciones si vas a trabajar con BigQuery

Ronny Medina — Mon, 09 Sep 2024 03:16:16 +0000

¡Hola a todos! Espero que se encuentren bien. En este post, quiero compartir mi experiencia trabajando con BigQuery y los aprendizajes que obtuve en el proceso.

Hace un tiempo, me enfrenté a la tarea de sincronizar los datos de una base de datos a BigQuery. El reto no solo era trasladar la información desde una base de datos relacional, sino hacerlo de manera que los datos estuvieran más procesados y optimizados para su uso en BigQuery

Cómo aprovechar BigQuery sin encasillarlo en el modelo relacional o no relacional

En este punto, lo que quiero decir es que no trates de llevar al 100% la misma estructura que tienes en tu base de datos relacional o no relacional. No quiero decir que esté mal; va a depender de cada caso. Desde mi perspectiva, intentar trasladar esta misma relación no genera ningún beneficio en BigQuery que sea significativo, o al menos no desde mi conocimiento, ya que algunas relaciones que hacemos son para poder optimizar consultas y evitar duplicar datos en tablas.

Como en la imagen anterior, tal vez no sea necesario intentar trasladar la tabla transaction_types, ya que esta solo sirve para evitar duplicar datos en una misma tabla.

BigQuery te permite trabajar con datos estructurados y no estructurados, y se define como un servicio de almacenamiento.

Puntos relevantes:

No soporta transacciones
No existen índices para hacer relaciones o tener campos únicos
Permite el uso de SQL para manejar operaciones
También puedes modelar los datos como si trabajaras en Mongo

Si puedes saber de antemano cómo se va a operar con los datos, esto te será de gran ayuda al momento de trasladar la información. En ocasiones, no es posible conocer cómo se manejarán los datos al 100%, ya que pueden ser equipos externos quienes los manipulen con otras operaciones.

Comprender los costos operativos de BigQuery

Es importante entender los costos operativos, ya que BigQuery aplica diferentes tipos de cobros según las operaciones que se realicen. Dependiendo de lo que se quiera llevar a cabo, es fundamental tener estos costos en cuenta, ya que pueden ayudar a optimizar las operaciones y evitar gastos innecesarios. Suponiendo que solo queremos almacenar, consultar y transferir los datos, podemos destacar lo siguiente:

Costo por almacenamiento

Costo por consultas, que incluye la definición de esquemas y la manipulación de datos

Costo por transferencia de datos

Te recomiendo consultar la página de precios de BigQuery para obtener información detallada y realizar un mejor análisis e investigación de los costos.

Puntos relevantes:

Tener presente los costos y saber como vas operar

Riesgos de la simplificación excesiva en la estructura de tablas

Como mencioné anteriormente sobre no trasladar directamente la estructura de tu modelo de base de datos a BigQuery, en el proceso de intentar simplificar o desnormalizar los datos, podríamos caer en el error de complicar el manejo de la información. Por ejemplo, en mi caso, tenía una situación similar, donde había una relación de uno a muchos.

more....

Configuración de Dockerfile para proyecto Nest.js: tutorial práctico

Ronny Medina — Sat, 22 Jun 2024 02:54:04 +0000

Post original .

Hola a todos, en este post quiero hacer una actualización de mi post anterior en Nest.js. Quiero dejar una configuración básica para un proyecto nuevo con este framework usando Docker.

Requerimientos

Tener Nodejs
Tener Docker
Conocimientos basicos en Docker

Instalación

El primer comando que necesitamos usar es el siguiente, que se encuentra en la página oficial, para poder instalar la CLI en nuestro sistema operativo y poder generar el proyecto.



$ npm i -g @nestjs/cli
$ nest new project-name

Creando nuestro Dockerfile

Lo primero que necesitamos es crear un archivo que se encargue de levantar nuestra app en modo productivo o desarrollo. Este archivo se llamará run.sh



#!/bin/bash
echo "NODE_ENV: $NODE_ENV";

if [ "$NODE_ENV" == "prod" ]
    then
        echo "start prod mode...";
        npm run start:prod
else
    echo "start dev mode...";
    npm run start:dev
fi

Este archivo lee la variable de entorno NODE_ENV y, en caso de ser el valor «prod», ejecutará el comando para correr en modo productivo; caso contrario, ejecutará nuestra app en modo desarrollo.

Luego de crear este archivo, vamos a crear nuestro archivo Dockerfile.



FROM node:20.14.0-bullseye

ENV APP_PORT 3000
ENV NODE_ENV prod
ENV WORKDIR_APP /var/prod

WORKDIR ${WORKDIR_APP}
COPY package.json .
RUN npm install
COPY . .

RUN npm run build

EXPOSE ${APP_PORT}

CMD ["bash", "run.sh"]

Configurando nuestra app

En nuestro proyecto vamos a crear una carpeta llamada config y dentro un archivo llamado envs.ts. Este archivo tendrá nuestras variables de entorno de nuestra aplicación.



export const APP_PORT = parseInt(process.env.APP_PORT, 10);
export const APP_IS_PROD = process.env.NODE_ENV === 'prod'

Luego vamos a configurar nuestro main.ts. Vamos a definir una variable llamada logLevel. Esta contendrá los niveles de log por defecto en modo productivo y agregaremos los otros en modo desarrollo. Esto es completamente configurable a sus necesidades.

Adicionalmente activaremos el versionamiento de nuestra API.




import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { APP_IS_PROD, APP_PORT } from './config/envs';
import { LogLevel, Logger, VersioningType } from '@nestjs/common';

async function bootstrap() {
  const logger = new Logger('bootstrap');
  const logLevel: LogLevel[] = ['error', 'warn', 'fatal', 'log'];

  if (!APP_IS_PROD) {
    logLevel.push('debug', 'verbose');  }


  const app = await NestFactory.create(AppModule, { logger: logLevel });

  app.enableVersioning({type: VersioningType.URI});

  await app.listen(APP_PORT);
}
bootstrap();

Construir nuestra imagen

Ahora podemos correr los siguientes comandos para construir nuestra imagen.




$ docker build -t nestjs-setup .
$ docker run -it --rm -e NODE_ENV=develop -p "3000:3000" --name nest-setup nest-setup

Ahora debemos poder acceder al puerto 3000 y ver el mensaje de Hello World!.

Agregando validaciones & Swagger

Adicionalmente voy a agregar los paquetes para las validaciones y para documentar nuestra API.

Para eso debemos entrar en nuestro contenedor que esta corriendo el siguiente comando:




$ docker exec -it nest-setup bash

Agregar las siguientes dependencias:

Swagger
Class validator




$ yarn add class-validator class-transformer
$ yarn add @nestjs/swagger

Para este punto, debemos actualizar nuestro package.json, ya que los cambios se realizaron dentro del contenedor y no estamos usando ningún volumen en este momento. Podemos copiar el archivo package.json desde nuestro contenedor usando docker cp. No es necesario que realice este paso, ya que al final del post tendrá todo el código con todas las dependencias.

Ahora en nuestro main vamos a agregar una configuración adicional para poder ver los errores de las validaciones cuando estemos en modo desarrollo. Este paso para ver los errores es opcional pero puede ser útil si desea aplicar otro caso de uso.




import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { SwaggerModule, DocumentBuilder } from '@nestjs/swagger';
import { APP_IS_PROD, APP_PORT } from './config/envs';
import { BadRequestException, LogLevel, Logger, ValidationError, ValidationPipe, VersioningType } from '@nestjs/common';

async function bootstrap() {
  const logger = new Logger('bootstrap');
  const logLevel: LogLevel[] = ['error', 'warn', 'fatal', 'log'];

  if (!APP_IS_PROD) {
    logLevel.push('debug', 'verbose');
  }

  const app = await NestFactory.create(AppModule, { logger: logLevel });

  app.enableVersioning({type: VersioningType.URI});

  const config = new DocumentBuilder()
    .setTitle('Cats example')
    .setDescription('The cats API description')
    .setVersion('1.0')
    .addTag('cats')
    .build();

  const document = SwaggerModule.createDocument(app, config);
  SwaggerModule.setup('docs', app, document);

   // print validation errors
   app.useGlobalPipes(
    new ValidationPipe({
      enableDebugMessages: !APP_IS_PROD,
      transform: true,
      exceptionFactory: (errors: ValidationError[]) => {
        const allErrorMsg = errors.flatMap((error) =>
          Object.values(error.constraints),
        );

        if (!APP_IS_PROD) {
          logger.debug('Validations errors', allErrorMsg);
          logger.debug('Error schema', errors);
        }

        return new BadRequestException(allErrorMsg);
      },
    }),
  );


  await app.listen(APP_PORT);
}
bootstrap();

Modificando nuestro app.controller.ts.




import { Body, Controller, Get, Post } from '@nestjs/common';
import { IsString } from 'class-validator';
import { ApiProperty } from '@nestjs/swagger';
import { AppService } from './app.service';

export class AppDto {
  @ApiProperty({ example: 'demo' })
  @IsString()
  name: string;
}

@Controller({
  version: '1',
  path: 'demo',
})
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Post()
  getHello(@Body() data: AppDto): string {
    return this.appService.getHello();
  }
}

Ahora podemos acceder a la siguiente URL: http://localhost:3000/docs. Al enviar la solicitud, podemos ver que los datos se envían correctamente.

Si enviamos un número al momento de realizar la solicitud, podemos ver los mensajes de error también en la terminal.

Docker compose

Por ultimo podemos agregar nuestro docker-compose.yml



version: '3'

services:

  nest-setup:

    container_name: nest-setup

    build: .

    ebvironment:

      - NODE_ENV=develop

    ports:

      - "3000:3000"

    volumes:

      - ./src:/var/prod/src

      - ./test:/var/prod/test

      - ./package.json:/var/prod/package.json



$ docker compose up

Recursos

Golang Logging Configuration with Zap: Practical Implementation Tips

Ronny Medina — Fri, 17 May 2024 14:03:14 +0000

Hi everybody, continuing from the previous post, which was a basic setup with Golang. In this part, we're setting up our logging in the app.

THE ORIGINAL POST HAS ADDITIONAL INFORMATION.

Installing the package zap

https://github.com/uber-go/zap

In our project, we need to install the zap package. To do this, put the following command in our terminal.

go get -u go.uber.org/zap

If you're using Docker like me, you need to enter the container. So, if you're running the app in the background, execute this command.

docker exec -it demogolang bash

To run the app in the background, use the command: docker-compose up -d.
The name 'demogolang' is declared inside the docker-compose.yml file.

Setting up the logging

Dockerfile

In our Dockerfile add 2 environment variables and created a folder to save the logs.

APP_LOG_LEVEL: The app log level could be debug or error. We use the debug level when we're in development and error when the app is in production mode. So, if you're in error mode when you print logs Info or Debug, these are not shown in the terminal. If you want to use more levels, you can edit the function to get the current level.
APP_LOG_FOLDER: This is the path where you place the log file. You can set whatever path you want.


ENV APP_LOG_LEVEL debug
ENV APP_LOG_FOLDER /tmp/logs/

RUN mkdir -p ${APP_LOG_FOLDER}

Reading environment variables

Inside our app folder, I created a folder called config. Inside config, I put the file called envs.go.


package config

import "os"

var APP_ENV = os.Getenv("APP_ENV")
var IS_DEVELOP_MODE = APP_ENV == "develop"
var APP_LOG_LEVEL = os.Getenv("APP_LOG_LEVEL")
var APP_LOG_FOLDER = os.Getenv("APP_LOG_FOLDER")

Logger file


package utils

import (
    "example/demogo/config"

    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

var Logger *zap.SugaredLogger

func getLevelLogger(level string) zapcore.Level {
    if level == "debug" {
        return zap.DebugLevel
    }

    return zap.ErrorLevel
}

func init() {
    var err error
    level := zap.NewAtomicLevelAt(getLevelLogger(config.APP_LOG_LEVEL))
    encoder := zap.NewProductionEncoderConfig()

    zapConfig := zap.NewProductionConfig()
    zapConfig.EncoderConfig = encoder
    zapConfig.Level = level
    zapConfig.Development = config.IS_DEVELOP_MODE
    zapConfig.Encoding = "json"
    zapConfig.InitialFields = map[string]interface{}{"idtx": "999"}
    zapConfig.OutputPaths = []string{"stdout", config.APP_LOG_FOLDER + "app_log.log"}
    zapConfig.ErrorOutputPaths = []string{"stderr"}
    logger, err := zapConfig.Build()

    if err != nil {
        panic(err)
    }

    Logger = logger.Sugar()
}

I created a global variable called Logger; this will be set when the package is initialized. The function getLevelLogger is used to get the current level from the environment variable.

The init function is automatically called before the main function.

encoder: Is the log format, how you want to apply the format.
InitialFields: The values that you want to always be in the log.
OutputPaths: The output of the file.
ErrorOutputPaths: To print error log in the terminal.

Finally our main.go.


package main

import (
    "example/demogo/utils"
    "time"
)

func main() {
    defer utils.Logger.Sync()

    for {
        time.Sleep(5 * time.Second)
        utils.Logger.Infoln("hello world")
    }
}

It's important to call defer utils.Logger.Sync() in the entry point of our app. Normally ensures that all buffered log messages are sent to their final destination if the application crashes.

docker-compose up -d --build

docker logs -f demogolang

Basic setup for projects with Golang

Ronny Medina — Tue, 14 May 2024 02:08:14 +0000

Post original in my blog.

PART 2

Hi everybody! In this post, I'm going to explain how to create a Golang project with Docker. Video en español en mi canal.

This repository contains the code.

Requirements

Docker installed
Basic knowledge with docker
Docker compose installed

Structure project

The structure for this project is very simple.

app: Contains all the files of our application.
main.go: Application entry point.
Dockerfile: Dockerfile to build our image.
docker-compose.yml: File to help build our image more easily.
go.mod: File to track your project dependencies.

your-folder-project
    app
       main.go
       run.sh
       go.mod
    Dockerfile
    docker-compose.yml

Dockerfile

FROM golang:1.22.3-bullseye

ENV APP_ENV prod
ENV WORDIR_APP /usr/src
ENV BINARY_NAME main
ENV BINARY_PATH ${WORDIR_APP}/${BINARY_NAME}

WORKDIR ${WORDIR_APP}
COPY app/go.mod app/run.sh ./

RUN go mod download && \
    go mod verify && \
    go install github.com/cosmtrek/air@v1.51.0

WORKDIR ${WORDIR_APP}/app

COPY ./app .

RUN go build -v -o ${BINARY_PATH}

CMD [ "bash", "run.sh" ]

In this file we define image version FROM golang:1.22.3-bullseye. Then there are 4 environments.

APP_ENV: This environment is used to manage the mode in which our application is running. We use values like: (prod, develop, test).
WORDIR_APP: It is the location when we are working.
BINARY_NAME: This is the name of our application when it is finally built. You can use any name that you want.
BINARY_PATH: This is the full path of our binary when the application is built.

On line 11, we're installing all dependencies in our project, and additionally, we add this library to enable live-reload functionality in our application.

I think that was the most important thing in Dockerfile.

go.mod

This file is to track dependencies in your project. You can check the basic tutorial.

module example/demogo

go 1.22.3

module: The name of your package.

go: Golang version. In this case this version must be the same one that we use in our Dockerfile

run.sh

#!/bin/bash

echo "=== Start app in (${APP_ENV}) ==="

env=${APP_ENV:-}

if [ "$env" = "prod" ]; then
    # exucute binary
    ${BINARY_PATH} 
else
    air --build.cmd "go build -v -o ${BINARY_PATH}" --build.bin "${BINARY_PATH}"
fi

In this file, we have a conditional to execute a command depending on whether APP_ENV is prod or develop. If APP_ENV is prod, it executes the binary that we previously compiled in our Dockerfile. Otherwise, we execute the air command to use live reload in our app.

docker-compose.yml

version: '3'
services:
  demogolang:
    build: .
    environment:
      - APP_ENV=develop
    volumes:
      - ./app:/usr/src/app

Start app

To start our application we only have to execute the following command: docker compose up.

https://dev.to/ronnymedina/query-large-amounts-of-data-from-bigquery-faster-with-golang-4j61

Query large amounts of data from BigQuery faster with Golang

Ronny Medina — Sun, 29 Oct 2023 01:31:23 +0000

Original post from Edulixir.

Hi every one, In this posts I going to show you, how enable storage api. With this option, you can get big data results faster. I was working with BigQuery and in some cases I need to get big data results.

I'm going to assume that you already have the Google project with BigQuery active. For this example I'm going to use public dataset. It doesn't matter what data set you are using.

For example, this table has 540584 records.

Code without storage api


package main

import (
    "context"
    "fmt"
    "time"

    "cloud.google.com/go/bigquery"
    "google.golang.org/api/iterator"
    "google.golang.org/api/option"
)

var client *bigquery.Client
var ProjectId string
var CREDENTIALS_DIR = "credential.json"

func timeTracker(start time.Time, name string) {
    elapsed := time.Since(start)
    msg := fmt.Sprintf("==== %s -> time execute: %s ===", name, elapsed)

    fmt.Println(msg)
}

func connectClientToBq(ctx context.Context, projectId string) error {
    var err error
    path := CREDENTIALS_DIR

    credentialOpts := option.WithCredentialsFile(path)
    client, err = bigquery.NewClient(ctx, projectId, credentialOpts)
    if err != nil {
        return err
    }

    return nil
}

func closeClientBq() {
    ProjectId = ""

    if client != nil {
        client.Close()
    }
}

func executeQueryWithJob(ctx context.Context, sql string, params []bigquery.QueryParameter) (*bigquery.RowIterator, error) {
    defer timeTracker(time.Now(), "ExecuteQueryWithJob")

    q := client.Query(sql)
    q.Parameters = params
    job, err := q.Run(ctx)

    if err != nil {
        return nil, err
    }

    status, err := job.Wait(ctx)
    if err != nil {
        return nil, err
    }

    if err := status.Err(); err != nil {
        return nil, err
    }

    return job.Read(ctx)
}

func main() {
    defer closeClientBq()
    defer timeTracker(time.Now(), "Finish")
    var params []bigquery.QueryParameter
    ctx := context.Background()
    projectId := "demos-403423"
    err := connectClientToBq(ctx, projectId)

    if err != nil {
        fmt.Println(err.Error())
    }

    sql := "SELECT difficultyTarget, version, work_terahash, block_id FROM `bigquery-public-data.bitcoin_blockchain.blocks`"
    data, err := executeQueryWithJob(ctx, sql, params)

    if err != nil {
        fmt.Println(err.Error())
    }

    for {
        var values []bigquery.Value
        err := data.Next(&values)

        if err == iterator.Done {
            break
        }
        if err != nil {
            fmt.Println(err.Error())
        }
    }
}

For this query the execution time was:

==== ExecuteQueryWithJob -> time execute: 1.939979125s ===

The execution time to get all the data was:

==== Finish -> time execute: 1m30.853989167s ===

Storage API

Now if you enable the storage API, in the function connectClientToBq

func connectClientToBq(ctx context.Context, projectId string) error {
    var err error
    path := CREDENTIALS_DIR

    credentialOpts := option.WithCredentialsFile(path)
    client, err = bigquery.NewClient(ctx, projectId, credentialOpts)
    if err != nil {
        return err
    }

    err = client.EnableStorageReadClient(ctx, credentialOpts)
    if err != nil {
        return err
    }

    return nil
}

The times now were:

==== ExecuteQueryWithJob -> time execute: 3.695876083s ===

The execution time to get all the data was:

==== Finish -> time execute: 23.907911666s ===

I hope this post is useful to you. If you can share

Resources

Geolocation with Postgresql+Nestjs+Typeorm

Ronny Medina — Fri, 03 Dec 2021 12:41:13 +0000

Hi everyone. It's been a long time since I last posted something.

Some time ago I needed create an application that using geolocation to find ads nearby.

In my search, I take the easy solution which was to use Firebase.

Integration was easy because it was a new project. I did some testing and this worked great. I don't have much experience with Firebase.

Some problems I had were the following:

Try modeling tables in Firebase collections.
Thinking about the cost.
Difficulty trying to query multiple related collections

If I had more experience with Firebase this would be easy for me, but it is not the case.

So I found another solution that I think is easy to integrate.

Geo postgresql. You can visit the original publication and see in more detail all the explanations.

I did an example with Nestjs.

You can download this repository and run docker-compose up -d, this should get the server up. And you can access url /docs.

I hope this publication is helpful to you. I made a video tutorial in Spanish.

Publish/Subscribe pattern example (Redis, Kafka)

Ronny Medina — Sun, 18 Jul 2021 13:48:59 +0000

Hi, in tutorial I'm goin to show you how to implement this pattern. I won't explain details about Kafka or Redis. Then main idea here is to show you how to create simple implementation with these technologies.

To this tutorial we going to use Docker and docker-compose is necessary if you want to follow this tutorial.

Explain

The publish/subscribe pattern is very easy to understand. The main idea is to has central point where all the applications will put the information and then it will be consumed by other applications. You can see the imagen below.

Depends on the technologies we're using there're some terms that will change. For example in Redis to classify our information they call it a channel. In Kafka they call it topic.

Topics or channels is the way to classify information. For example, we will save information about the login process. We can create a (topic|chennel) called login_info and another to save then transaction information of a payment called transaction_payment.

Before start we need to install some dependencies.

npm install express ioredis kafkajs

docker-compose (Redis)

We need to create a docker-compose.yml and put this content.


version: "3.9"
services:
  redis-service:
    image: "redis:alpine"
    container_name: redis-service
    ports:
      - "6379:6379"

Publisher (Redis)

publisher.js. You can create an endpoint to send the information at moment. For me this approach works.

const express = require('express')
const Redis = require('ioredis')

const app = express()
const port = 3000
const redis = new Redis()

app.listen(port, async () => {
  const message = { message: 'message from redis' }
  redis.publish('test', JSON.stringify(message))
  console.log(`Example app (publisher) at http://localhost:${port}`)
})

Subscriber (Redis)

subscriber.js


const express = require('express')
const Redis = require('ioredis')

const app = express()
const port = 3500

const redis = new Redis()

app.listen(port, async () => {
  await redis.subscribe('test')

  redis.on('message', (channel, message) => {
    console.log(`Received ${message} from ${channel} (Redis)`)
  })

  console.log(`Example app (subscribe) at http://localhost:${port}`)
})

Now need to execute the following commands.


$ docker-compose up -d

$ node subscriber.js

$ node publisher.js // open in another terminal

You should now see a message in your terminal (subscriber). That was very easy =). Now we can do the same process to implement Kafka.

docker-compose (Kafka)

To implement Kafka we need to download a docker-compose.yml file from confluent page https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html.

We can use the entire docker-compose file with all services. But it is not necessary for this tutorial. The fundamental services to start working with Kafka are zookeeper and broker.


version: "3.9"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.2.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

Publisher (Kafka)


const express = require('express')
const { Kafka } = require('kafkajs')

const app = express()
const port = 3000


const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
})

app.listen(port, () => {
  const producer = kafka.producer()
  await producer.connect()
  await producer.send({
    topic: 'test',
    messages: [{ value: 'Hello KafkaJS user!' }],
  })

  await producer.disconnect()

  console.log(`Example app (publisher) at http://localhost:${port}`)
})

Subscriber (Kafka)

Before I begin, I need to clarify that this code could be anywhere you want.


  const admin = kafka.admin()

  try {
    await admin.connect()

    await admin.createTopics({
      waitForLeaders: true,
      topics: [{topic: 'test' }],
    })
  } catch(err) {
    console.error(err)
  } finally {
    await admin.disconnect()
  }

Kafka needs setup before starting. For example we need to create the topics before. Kafka has more settings unlike Redis. I put this content inside subscriber.js if you run this code multiple times this throws an error because the topic already exists.


const express = require('express')
const { Kafka } = require('kafkajs')

const app = express()
const port = 3500

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
})

app.listen(port, async () => {
  const admin = kafka.admin()

  try {
    await admin.connect()

    await admin.createTopics({
      waitForLeaders: true,
      topics: [{topic: 'test' }],
    })
  } catch(err) {
    console.error(err)
  } finally {
    await admin.disconnect()
  }

  // kafka
  const consumer = kafka.consumer({ groupId: 'test-group' })
  await consumer.connect()
  await consumer.subscribe({ topic: 'test', fromBeginning: true })
  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log(`Received ${message.value.toString()} from channel ${topic} (Kafka)`)
    },
  })

  console.log(`Example app (subscribe) at http://localhost:${port}`)
})

That's it to be able to work with Kafka. Now we can run the same commands we saw before.


$ docker-compose up -d

$ node subscriber.js

$ node publisher.js // open in another terminal

I hope it was useful for you and that you can start working with these technologies. If you have any corrections, let me know.

FastApi Example

Ronny Medina — Mon, 14 Jun 2021 02:35:28 +0000

Video en español.

Hello everyone, in this post I'm going to show you a small example with FastApi.

Well, to use FastApi, we need to install some dependencies such as:

pip install fastapi
pip install uvicorn[standard]

Or we can create a requirements file. requirements.txt.

fastapi==0.65.2
uvicorn==0.14.0

And then run the following command pip install -r requirements.txt.

The next step is to create a main file main.py and put the following content inside.

from fastapi import FastAPI

app = FastAPI()


@app.get('/')
def root():
    return 'ok'


@app.get('/item/{id}')
def read_item(id: int):
    return id

Now we can run the following command uvicorn main:app --reload.

We can check the urls in our browser http://localhost:8000/ and http://localhost:8000/item/1.

By default, FastApi has swagger included. We can check this url in your browser http://localhost:8000/docs. Another system is also integrated to document our api. You can visit http://localhost:8000/redoc.

The documentation of our api is automatic. This happens when we write the typing of our code. If you change this line def read_item(id: int) to def read_item(id: str) this updates our documentation.

To keep things in order we can create folder app and inside the following structure.

And inside the schemas we are going to create two files. A item.py this file is to save the validations of this resource.

from typing import Optional
from pydantic import BaseModel, Field

class Item(BaseModel):
    name: str =  Field(None, title='Example title...', min_length=3, max_length=10)
    price: float = Field(None, title='Price 300')
    is_offer: Optional[bool] = Field(False, title='Is offer')

You can visit the official page for more information pydantic-docs. Then we are going to create a __init__.py to export this validation.


from .item import Item as ItemSchema

Now we're going to create router file (router.py).


from typing import List
from fastapi import APIRouter

from .schemas import ItemSchema


router = APIRouter()

@router.get('/items', tags=['items'], response_model=List[ItemSchema])
def list_item() -> List[ItemSchema]:
    return []


@router.get('/items/{id}', tags=['items'])
def read_item(id: int) -> int:
    return id


@router.post('/items', tags=['items'], response_model=ItemSchema)
def create_item(data: ItemSchema):
    return data

Now we need to export this router to use it. You can create another __init__.py at the same level as the item folder.


from .router import router as item_router

And finally we need to update our main file.


from fastapi import FastAPI

from .items import item_router

app = FastAPI()

app.include_router(item_router)


@app.get('/')
def root():
    return 'ok'

To run our api we can execute this command uvicorn app.main:app --reload

Now we split our application and update the documentation.

How does validation work in FastApi?

This runs as a middleware if the data is invalid the return statement is never executed. You can try to pass invalid data to this API.


@router.post('/items', tags=['items'], response_model=ItemSchema)
def create_item(data: ItemSchema):
    return data

Well this was a small example with FastApi. I Hope this was helpful to you.

How to split your React app?

Ronny Medina — Mon, 05 Apr 2021 14:41:18 +0000

Hi everybody in this post I'm going to show you, how to split your React app. This post is very easy and quick to follow.

Why do you need to split your application?

To answer this question in two phrases, can be:

Gain performance in your application
Don't load unnecessary content

You could see this example (Before):

In that example, we loaded these icons asynchronously. Sometimes we have very heavy assets.

import React from 'react';

import DeleteIcon from '@material-ui/icons/Delete';
import DeleteOutlinedIcon from '@material-ui/icons/DeleteOutlined';
import DeleteRoundedIcon from '@material-ui/icons/DeleteRounded';
import DeleteTwoToneIcon from '@material-ui/icons/DeleteTwoTone';
import DeleteSharpIcon from '@material-ui/icons/DeleteSharp';
import DeleteForeverIcon from '@material-ui/icons/DeleteForever';
import DeleteForeverOutlinedIcon from '@material-ui/icons/DeleteForeverOutlined';
import DeleteForeverRoundedIcon from '@material-ui/icons/DeleteForeverRounded';
import DeleteForeverTwoToneIcon from '@material-ui/icons/DeleteForeverTwoTone';
import DeleteForeverSharpIcon from '@material-ui/icons/DeleteForeverSharp';
import ThreeDRotationIcon from '@material-ui/icons/ThreeDRotation';
import FourKIcon from '@material-ui/icons/FourK';
import ThreeSixtyIcon from '@material-ui/icons/ThreeSixty';

export const Home = () => {
  return (
    <div>
      <h1>Home</h1>
        <DeleteIcon />
        <DeleteOutlinedIcon />
        <DeleteRoundedIcon />
        <DeleteTwoToneIcon />
        <DeleteSharpIcon />
        <DeleteForeverIcon />
        <DeleteForeverOutlinedIcon />
        <DeleteForeverRoundedIcon />
        <DeleteForeverTwoToneIcon />
        <DeleteForeverSharpIcon />
        <ThreeDRotationIcon />
        <FourKIcon />
        <ThreeSixtyIcon />
    </div>
  )
}

As we can see, our page loaded in 19.47 s. In the example above, our page is blocked until all content is loaded.

Using `React.lazy`

import React, { Suspense } from 'react';

const DeleteIcon = React.lazy(() => import('@material-ui/icons/Delete'));
const DeleteOutlinedIcon = React.lazy(() => import('@material-ui/icons/DeleteOutlined'));
const DeleteRoundedIcon = React.lazy(() => import('@material-ui/icons/DeleteRounded'));
const DeleteTwoToneIcon = React.lazy(() => import('@material-ui/icons/DeleteTwoTone'));
const DeleteSharpIcon = React.lazy(() => import('@material-ui/icons/DeleteSharp'));
const DeleteForeverIcon = React.lazy(() => import('@material-ui/icons/DeleteForever'));
const DeleteForeverOutlinedIcon = React.lazy(() => import('@material-ui/icons/DeleteForeverOutlined'));
const DeleteForeverRoundedIcon = React.lazy(() => import('@material-ui/icons/DeleteForeverRounded'));
const DeleteForeverTwoToneIcon = React.lazy(() => import('@material-ui/icons/DeleteForeverTwoTone'));
const DeleteForeverSharpIcon = React.lazy(() => import('@material-ui/icons/DeleteForeverSharp'));
const ThreeDRotationIcon = React.lazy(() => import('@material-ui/icons/ThreeDRotation'));
const FourKIcon = React.lazy(() => import('@material-ui/icons/FourK'));
const ThreeSixtyIcon = React.lazy(() => import('@material-ui/icons/ThreeSixty'));


export const Home = () => {
  return (
    <div>
      <h1>Home</h1>
      <Suspense fallback={<div>Loading...</div>}>
        <DeleteIcon />
        <DeleteOutlinedIcon />
        <DeleteRoundedIcon />
        <DeleteTwoToneIcon />
        <DeleteSharpIcon />
        <DeleteForeverIcon />
        <DeleteForeverOutlinedIcon />
        <DeleteForeverRoundedIcon />
        <DeleteForeverTwoToneIcon />
        <DeleteForeverSharpIcon />
        <ThreeDRotationIcon />
        <FourKIcon />
        <ThreeSixtyIcon />
        </Suspense>
    </div>
  )
}

Now we can see the Home text before all the content is loaded (22.66s).

Our page takes a little longer to load but that's ok.

We can imagine that we have a list of users and two buttons. One button opens modal to add a new user and the other button is another modal to send feedback. So we can use React.lazy to load these modals. We can say that the main functionality is our user list. We don't need to lock the whole page for two modals that maybe the user does not use.

Split our routes

We can split routes, look at this example.

import React, { Suspense, lazy } from 'react';
import { BrowserRouter as Router, Route, Switch } from 'react-router-dom';
import Skeleton from '@material-ui/lab/Skeleton';

const Home = lazy(() =>
 import(/* webpackChunkName: 'Home' */'./components/home').then(m => ({ default: m.Home }))
);

function App() {
  return (
    <div className="App">
      <Router>
        <Suspense fallback={<Skeleton variant="rect" width={210} height={118} />}>
          <Switch>
            <Route exact path="/" component={Home}/>
          </Switch>
        </Suspense>
      </Router>
    </div>
  );
}

export default App;

I hope this post is helpful to you. If you have corrections please write me a comment. Thank you very much =)

Kafka introducción & implementación en Nodejs (node+express)

Ronny Medina — Mon, 01 Mar 2021 02:33:09 +0000

En este ejemplo veremos como implementar kafka y como podemos escribir y leer mensajes con nodejs. Puedes ver el video para tener información más detallada.

Código del ejemplo

ronnymedina / kafka-nodejs

kafka-nodejs

video

View on GitHub

Tutorial

Resources for learning data structures

Ronny Medina — Mon, 08 Feb 2021 13:08:02 +0000

Hello everyone, in this post I show you some resources to learn about data structure and improve your skills.

geeksforgeeks. I like this page, because it has many resources for learn, about data structures, algorithms, languages and more.
Introduction to data structures
. This is a video tutorial about data strucutres, the complete course is c++.

Some tips to improve our team work

Ronny Medina — Mon, 08 Feb 2021 13:04:22 +0000

Hi everyone, in this post I'm going to recommend some tips to improve as a real team.

Request feedback

This is important because we can improve our skills such as the tasks you've been assigned.

For example you need to create a service to create customers and send the email to them when it's created. You can get feedback of some members of our team about this feature. This shows a good impression of you, because this shows that you trust in our team.

Other examples could be that you need to do Pub/Sub service and you don't have much experience with that, so if you get feedback of some members of our team, together we can find a better solution or avoid some mistakes previously. To get feedback is important because it allows us to improve and gain confidence with our team.

Drive your team to improve

A good idea to encourage your team to improve is to do some activities to learn new things related to their jobs.

I remember that in a former job I made two activities, These activities were:

Every two weeks each member must show some technologies or good practices about code or whatever they wanted to show. I liked this activity, because I learned different things that I didn't know and it was a good strategy to balance the knowledge in our team.
The second activity was competitive programming. For that I created an activity in the calendar and sent the invitation to each participant. This activity didn't last too long for reasons of time but it was funny.
Another activity that I saw in several companies was "Game Space", I like it very much because it allows us to meet our partners beyond the workplace.

Be nice

It's important to be nice with our partners when your are sending feedback o when You are correcting a mistake made by a colleague, if you aren't nice with our partners this could generate inconvenience with them. This is not always so easy but we can do our best. We will not always agree with the ideas of others or they will not agree with our ideas. Always remember to be nice to all of them.

These tips may be relevant or irrelevant to you, depending on your current case. The most important thing here is to create a good place to work.

DEV Community: Ronny Medina

Recomendaciones si vas a trabajar con BigQuery

Cómo aprovechar BigQuery sin encasillarlo en el modelo relacional o no relacional

Comprender los costos operativos de BigQuery

Riesgos de la simplificación excesiva en la estructura de tablas

Configuración de Dockerfile para proyecto Nest.js: tutorial práctico

Requerimientos

Instalación

Creando nuestro Dockerfile

Configurando nuestra app

Construir nuestra imagen

Agregando validaciones & Swagger

Docker compose

Recursos

Golang Logging Configuration with Zap: Practical Implementation Tips

Installing the package zap

Setting up the logging

Dockerfile

Reading environment variables

Logger file

Basic setup for projects with Golang

Requirements

Structure project

Dockerfile

go.mod

run.sh

docker-compose.yml

Start app

RELATED

Query large amounts of data from BigQuery faster with Golang

Code without storage api

Storage API

Resources

Geolocation with Postgresql+Nestjs+Typeorm

Publish/Subscribe pattern example (Redis, Kafka)

Explain

docker-compose (Redis)

Publisher (Redis)

Subscriber (Redis)

docker-compose (Kafka)

Publisher (Kafka)

Subscriber (Kafka)

FastApi Example

How does validation work in FastApi?

How to split your React app?

Why do you need to split your application?

Using React.lazy

Split our routes

Kafka introducción & implementación en Nodejs (node+express)

Código del ejemplo

ronnymedina / kafka-nodejs

kafka-nodejs

Tutorial

Resources for learning data structures

Some tips to improve our team work

Request feedback

Drive your team to improve

Be nice

Using `React.lazy`