DEV Community: Nicolas Lima

Testing Apollo Server Graph API

Nicolas Lima — Thu, 30 Apr 2020 18:06:36 +0000

I often see people on community complaining about testing Graph API's, So I decided to create this little template using apollo server and jest with Typescript! I hope you can use it as a base to make something just as good as you need! The idea is to separate the application from the server, this way you can use supertest npm module to make real requests and do not need to initiate a local server!

https://github.com/deLimaNicolas/graphql-typescript-test-boilerplate

Clickhouse over Postgresql?

Nicolas Lima — Sat, 11 Jan 2020 22:17:51 +0000

About this post

All right, fellows, I'm quite new to the data world, my entire career was development itself, but aren't we always trying to get better? That's why I'm taking my chances now, and my first challenge in my adventure is to decide between Postgres and Clickhouse (I'll explain at the problem section). Anyway, you can use this article as a source of knowledge but I beg you to don't skip the comments section because the main point of this writing is to help me discover all concept mistakes I have in my mind! So read this, but go check what te good coders have to say about it :).

The problem

Now, the most important part of every task, understanding the problem.

Although I need to store data for transactions and analysis, two different things, I still have all my application info stored in one Postgres DB. I guess for an app with a small amount of data that would be nothing to worry about, I mean, Postgres was not made for OLAP but it can handle the job. But in my case scenario, I have a Bigdata challenge, we're saving a considerable quantity of historical data along with daily transactional info and this is inflicting our performance.

Another worthy thing to remember about my problem: The end-user can create his own queries, so it's a little bit difficult to predict how our DB it's gonna be used.

The solution

Here is where I expose myself, I thought about some workarounds and the point is for you guys read and tell me when and if I got it wrong, if you agree and got extra tips or own experience stories, please tell me all about! Appreciate!

With Postgres

I need to, at least, try to maintain Postgres in my system, after all, it seems easier when you compare to migrate your entire application to Clickhouse. Let's skip obvious things, such as updating hardware, isolating DB from application etc.

The only way I figured out to continue with Postgres is by changing our system architecture. Since is quite impossible to predict all types of queries our users will create, indexing and SQL structures will not solve 100% of the problem. That's why I'm thinking about an approach where we store historical and transactional data into two different environments, this way when our end-user create his own query it wouldn't need to scan an entire big table, only a small piece of it and perhaps if he needs to access the oldest data we can redirect him to the other slower environment.
But this will apply more complexity to our application and it goes against a hybrid OLAP/OLTP architecture, that I believe is a good bet to the future!
Any tips here?

Clickhouse

If you don't know much about Clickhouse, please check this link, in short ClickHouse is a columnar DBMS for OLAP, it is great to run queries on a big amount of data, here you can see a benchmark comparison between Postgres and Clickhouse.
Clickhouse itself would easily solve our performance problem, bigdata is definitely not a problem for this guy, but still, not 100% good.

"Specialized tools are specialized: just remember the limitations! - Bad with heterogeneous hardware (Cloudflare experience) - Non-throttled recovery (source replicas flooded with replication load) - No real delete/update support, and no transactions - No secondary keys - Own protocol (no MySQL protocol support) - Limited SQL support, and the joins implementation is different. If you are migrating from MySQL or Spark, you will probably have to re-write all queries with joins" - hnmullany from Hackernews

All right, we kinda have a solution here, Clickhouse is for sure a good choice, but will not solve everything, that's the point when I started to think about combining the pros of Postgres and Clickhouse. Maybe a Foreign Data Wrapper could save me here. This way I can use Clickhouse to handle all massive data queries, even those that my end-user will create himself and for operations like delete and update I can continue to use Postgres features.

Using clickhousedb_fdw may be a good choice for that, but... Is not perfect as well, I don't have much experience using an FDW but I guess it would apply complexity to our application, mainly when we're talking about synching our foreign tables.

That's what I have so far, what you guys think about?

Thanks for stopping by! Good Luck and Good Code.

Building a Serverless REST API using MongoDB(mongoose), NodeJs and Serverless framework

Nicolas Lima — Tue, 01 May 2018 20:32:03 +0000

Goals

Build and test locally a Serverless API, that can access a local MongoDB using mongoose

Requirements

All right fellows, first I need to explain the reason that I created this post. I'm a young BackEnd developer, and I've encountered a lot of information about the creation of Serverless API's, but I realized a great lack of articles when it comes to build using mongoose and testing locally. Anyway, this is a quite simple tutorial, but if you're just like I was yesterday, maybe it can save you some hours.

This lesson is specific for people who already know almost everything to set up this API, but is suffering because of two or three poorly explained things. So, if you don't know how to set up your AWS account, or don't know how to use the serverless framework, here are some videos that helped me:
How to create and deploy a simple Serverless API
Understanding a little bit more about AWS LAMBDA
Installing MongoDB

Let's go to work!

Ok, So let's start installing the Serverless package:

npm install -g serverless

Now let's create a default service using the aws-nodejs template

sls create -t aws-nodejs -p api && cd api

Now you can see inside the folder we just created named 'api', that we have 2 files:'handler.js' and 'serverless.yml'

Now, inside your api folder, let's initialize npm

npm init

Now, let's install mongoose

sudo npm install mongoose --save

Ok guys, above, in the "Requirements" session, I recommended this Video, it will explain how the basic function inside our file works, re-watch if you feel needs, because we're gonna apply some changes on those files.

First, make sure you run "mongod" command, because we need a local MongoDB service.

Okay... Now, let's create a new folder inside our api directory, let's name it "Schemas", inside it we're gonna create a new Schema for our MongoDB, if you're not used to work with schemas, check this mongoose Doc about them

(inside our directory named 'api') mkdir schemas

Then

(inside schemas) touch user.js

Using some editor(I like Atom), edit user.js structure inside a var and export a module with this one

All right, now inside "handler.js" let's modify the same "hello" function to connect with our running Mongod service and execute a simple query, but before we need to import "mongoose", start a connection, get our userSchema and create a model.

(inside handler.js) var mongoose = require("mongoose");

Then

var connectorMongodb =  mongoose.connect('mongodb://localhost/mynewDB');

Almost there...

var userSchema = require('../api/schemas/userSchema.js');

Okay...

var userModel = mongoose.model('user',userSchema,'user');

Just to remember you guys, this lesson isn't for total beginners, so if you don't know how to use mongoose and what is a model or schema, go work on it first. This lesson is specific for people that are getting into trouble because of the lack of information about create a Serverles Rest API using specifically mongoose.

Ok, now let's apply some changes inside our "handler.js", and remember that we must close our mongoose connection once it was open, otherwise our function will "timeout" (Lambda functions have a default amount of seconds until it stops)

Now, inside your api directory, run this command in order to test your function "hello" locally

sls invoke local --function hello

It should returns you a Json with your user's collection information

I guess you should already have some user data stored inside your DB, just go to mongoshell and use db.user.insert

I hope i helped you guys, see you later.

My Instagram if you like to discuss anything with me