DEV Community: Leonardo Teteo

What is a good documentation for you and what tools do you use?

Leonardo Teteo — Wed, 15 Apr 2020 01:37:52 +0000

It seems documentation never gets the love its deserve for our own good and there are many tools for that, although I'm still looking for the right one.

What are the ones that make part of you guys pipelines?

Thoughts about quarantine and remote working after 15 days

Leonardo Teteo — Tue, 07 Apr 2020 21:18:45 +0000

With the crisis of the new coronavirus (COVID-19), it is never enough nowadays the recommendation to stay at home as much as possible to avoid infecting yourself and others. Here in Brazil the spread of the virus started at the beginning of March. Measures were taken immediately to prevent infection, soap and hand sanitizer flew from the shelves and it is hard to find them even now. A lock-down with only the extremely necessary opened was announced around March 18th, but the company I work advised us wisely to stay at home from Monday that week, March 16th, and I've been at home ever since.

The situation is serious and we all need to take it seriously, but I'm trying to stay positive, principally when it comes to remote working. There are people who really need to go out, meet people, talk face to face, etc. However, it is not like that for the majority of introverts like me. It is not like we don't like to go out at all, but it is not surprising that we generally like to stay at home more than other people, it is inside of our comfort zone. But anyway, I will stop generalizing and talk about my experience exclusively.

Before this crisis I worked from home once per week and since I started doing that I found myself much more productive at home than at the office. Besides staying in my comfort zone, noises and interruptions are less likely. Besides that, I can sleep more and decrease my 2 hours in public transportation to zero! Yes, public transportation around here are cramped and slow, it is not an exaggeration to say that I spend 50% of my energy or more going to and from work. Programming is what I like to do, even in my time off, so at work is more fun than stress (I'm not saying that there is no stressful moments, there is! But often I face them as a challenge), so yes, I spend more energy on my way to work than working. So, at home I'm always 100% after a peaceful breakfast before starting.

There are some interesting things that I noticed during my work remotely and even reading about remote working these days that I find interesting on how it is different and how it is advantageous to me in many ways. Some days ago I read a blog post by Martin Fowler on how to do effective video calls. At my company we don't generally do video calls, we keep with just audio, although the meeting app has this feature, of course. However, some rules apply also to audio calls, principally two that go well together: Mute when you are not talking and Don't talk over people. This is HUGE for an introvert like me. People may think it is an exaggeration, but there are differences in tone of voice and how people behave in meetings and all that. Extroverts are generally the center of attention, they talk a lot and loud, in live meetings people have more freedom to talk over one another and still being understood. People who are introverts, or who are not even introverts, but just have a low tone of voice, are generally overwhelmed.

On video/audio calls the microfones are not that forgiving, people talking over one another is a complete mess, you don't understand anything, so people have to speak one at a time. The participants have to mute and unmute all the time, this also slows the reactions and things get more organized. Also, people tend to be more concise and avoid small talks. In the end, meetings get shorter, more productive and more people are heard. It is a win-win situation in my point of view.

Overall it has been a great experience for me, but I'm aware that this is an extreme case and if things keep that way for long, it will start to get a little boring. Even when you work remotely, sometimes you go out of your house to work at a cafe your something, to change the scenario, this is impossible right now. Let's hope that we can win this battle against the virus soon and companies improve their remote working policies after this great, but unintended, experiment!

Last, but not least, we have to remember the privilege that we have by being able to work from home, there are people who don't have this privilege and are suffering greatly and we need to put them on our thoughts and prayers. We have this privilege, but with it comes great responsibilities. Now than ever before the technology is connecting people, making them see they are not alone even in this crisis. We from IT are working to keep everything running: messaging apps, streaming apps, news sites, social media, etc. I'm proud and humbled to be able to be part of this great team that is keeping everything running so that people stay connected and entertained in the middle of this crisis. We are going to defeat this virus together! :)

Building an URL Shortening API with Spring WebFlux (and a lot of supporting cast)

Leonardo Teteo — Sat, 04 Apr 2020 15:12:20 +0000

Introduction

In the past few weeks and specially the last few days with the recommendation to stay at home due to the pandemic crisis we are fighting against (stay at home, guys! it is never an exaggeration this recommendation), I've been reading a lot of articles and experimenting with new technologies. On one of those occasions I came across this great article by Sunil P V about how a URL shortening application works and after reading it and seeing the architecture I thought that developing it would be a great way to pass time. I wrote before about how I struggle at having ideas for practice projects here and here, so it sounded reasonable.

Explaining the Architecture

Borrowing the diagram that Sunil P V has in his article I will explain the technologies I used to develop the project and why. First, let's see the diagram:

Every URL shortening application I've seen so far works similarly. When the user sends a long URL, the API generates a short hash that identifies the long URL and saves both information into a database, then when the user requests for sho.rt/hash the API gets the hash and the long URL represented by it and redirects to that URL. Easy enough.

The most important part of the architecture is the API itself, represented by the green box in the diagram, this API is supposed to have many instances behind a load balancer, which means it needs to be distributed. We have the database where the URL information is persisted, Redis for caching purposes and Zookeeper for coordination. Let's go into more details:

URL Shortening API: The API needs to be distributed and resilient. Since it was a project made to practice new things, I mixed things I know (Spring) and a subject I wanted to experiment for some time (WebFlux), and so the application is made with Spring WebFlux as its base. A small load test done with Apache JMeter suggests that thanks to its non-blocking approach it is much more resilient than plain old Spring MVC in high load. In addition to that, the architecture suggests the use of the library Hashids to generate the hashes for the URL, I followed this recommendation, more on that later.
Database: The database described by Sunil P V in this article is DynamoDB, but it also mentions relational databases. I decided to use a relational database for the easy setup (no need for cloud configuration, etc.) and because it also offered me another opportunity to experiment. Since the application itself was reactive and non-blocking, the remaining of the components must be as well to take full advantage of its benefits. As such, I decided to use PostgreSQL with its reactive driver and powered by Spring R2DBC. Spring R2DBC makes the use of its repositories a smooth sailing for anybody who used Spring Data JPA before, so it was not that new, but it is great that both controller and data layers talk the same language with Mono and Flux all over.
Redis: This is the easiest part to explain, not only because it is recommended in the architecture, but also because it is almost the standard for caching nowadays. It is used to cache the short url x long url relation so that it doesn't burden the database with numerous GETs for the same URL, basic caching. Spring also offers a reactive implementation for Redis, so it is a great idea to use it into the architecture.
Zookeeper: Maybe this is an overhead, although the article mentions Zookeeper specifically, it also says that it could be replaced by Redis. Zookeeper is just used to store a counter that it is shared by all instances of the API, which is used to generate the range of ids a given instance will be responsible for. Let me know what you think about this, since it is something that's new for me. First time ever I used Zookeeper.

The code can be found here and it also has some optional features that I was experimenting with such as tracing with Zipkin and metrics export to Elasticsearch. Since it does not make part of the core, I will not talk about it here, but feel free to enable them if you want.

Code Details

Lets start the code discussion by the specifics of the configuration. There are two very important beans and one for convenience that are configured. For convenience I declare a Hashids bean with the salt already so that I don't need to recreate every time I use:

@Bean
public Hashids hashids() {
  return new Hashids(hashIdsSalt);
}

Then there is a pretty straightforward Redis configuration, but notice that it is Reactive. Spring Data Redis already have a connection factory ready to use, I just need to have a ReactiveRedisTemplate to use:

@Bean
public ReactiveRedisOperations<String, Object> redisOperations(ReactiveRedisConnectionFactory connectionFactory) {
    GenericJackson2JsonRedisSerializer serializer = new GenericJackson2JsonRedisSerializer();
    RedisSerializationContext<String, Object> context =
    RedisSerializationContext.<String, Object>newSerializationContext(new StringRedisSerializer())
            .value(serializer).build();
    return new ReactiveRedisTemplate<>(connectionFactory, context);
}

Then there is a bean called UrlIdRange that is used to control the range of ids that the instance of the application is responsible. As I said before, the range is configured on Zookeeper, but every instance has its own range of ids to assign and only when it is exhausted it goes to the Zookeeper server again to get a new range. It already contacts Zookeeper to create the range on bean creation.

@Bean
public UrlIdRange urlIdRange(SharedConfigurationService sharedConfigurationService) {
    Integer counter = sharedConfigurationService.getSharedCounter(urlRangeKey);
    return new UrlIdRange(counter);
}

With that it is a good time to explain the SharedConfigurationService and the UrlIdRange classes.

The implementation of SharedConfigurationService uses Apache Curator Framework to connect to the Zookeeper server and perform the actions on the shared counter. I will spend a little more time on this because it is something I did for the first time and suggestions on how to improve or even say that something is wrong is welcomed. The class only has one method getSharedCounter and a property that gets the Zookeeper server base url from the application properties yml file. It receives a key of type String and then starts its processing:

1 - Starts the client:

final CuratorFramework client = newClient(baseUrl,new RetryNTimes(3, 100));
client.start();

2 - Create the SharedCount:

SharedCount sharedCounter = new SharedCount(client, key, 0);

3 - On try-catch block, everything is done:

try {
  sharedCounter.start();

  VersionedValue<Integer> counter = sharedCounter.getVersionedValue();
  while(!sharedCounter.trySetCount(counter, counter.getValue() + 1)) {
      counter = sharedCounter.getVersionedValue();
  }

  sharedCounter.close();
  client.close();
  return counter.getValue();
} catch (Exception e) {
  log.error("Error while starting shared counter, impossible to update counter.", e);
  throw new NotAbleToUpdateCounterException();
}

The SharedCount is started and then the application get its versioned value and try to set the new value for the counter, it will only set if the values hasn't changed, otherwise it will keep trying. I did this way because I was thinking about the distributed characteristic of the application, in a high load environment collisions could happen and it would be a trouble for the management of the ids. I don't know if it is good practice or even if it is really necessary, so give me a heads up about what could go wrong or not. Then I close the counter and the client and return the value.

UrlIdRange constructor is called on the bean creation with the current counter and then it does the following:

public UrlIdRange(Integer counter) {
    this.calculateRange(counter);
    this.hasNext = true;
}

public void calculateRange(Integer counter) {
    this.initialValue = counter * 100_000;
    this.currentValue = new AtomicInteger(initialValue);
    this.finalValue = initialValue + 99_999;
    this.hasNext = true;
}

The constructor calls calculateRange where the calculation is done, the range is 100.000 ids, if the counter is 0, then it will be 0 to 99.999 and so on. Maybe I exaggerated a little bit on the number and it could be configurable through .yml file.

With the initial configuration out of the way, it is time to dive into WebFlux. The API have two endpoints, as I've said before, one GET to find the URL and redirect and a POST to generate new short URLs. The router is very simple:

@Bean
public RouterFunction<ServerResponse> route(UrlInfoHandler handler, ErrorHandler errorHandler) {
    return RouterFunctions.route()
            .onError(Exception.class, errorHandler::handleError)
            .GET("/{url}", handler::findLongUrlAndRedirect)
            .POST("/", accept(MediaType.APPLICATION_JSON), handler::generateAndSaveShortUrl)
            .build();
}

People may find the @RequestMapping annotation from Spring MVC convenient, but I think a router function like this where all routes are defined in one place also has its advantages, it is much easier to see what routes are available and Spring gives some features to keep it organized in large applications. Since Spring Boot is made for microservices, which shouldn't have many routes per application anyway, it is not a problem.

Then we have the UrlInfoHandler where the fun happens. It is very interesting and maybe a bit overwhelming to face declarative programming for the first time, but since I had some experience already with Javascript and its declarative, non-blocking style, it was not so bad, but certainly I need to show to some of colleagues later to see if they understand what is happening. Let's start with the findLongUrlAndRedirect method:

public Mono<ServerResponse> findLongUrlAndRedirect(ServerRequest serverRequest) {
  String shortUrl = serverRequest.pathVariable("url");
  return cacheService.getFromCacheOrSupplier(shortUrl, UrlInfo.class, () -> urlInfoService.findByShortUrl(shortUrl))
          .doOnNext(urlInfo -> log.info("UrlInfo found: {}", urlInfo))
          .flatMap(urlInfo -> permanentRedirect(URI.create(urlInfo.getLongUrl())).build())
          .switchIfEmpty(status(HttpStatus.NOT_FOUND)
                  .bodyValue(ErrorResponse.notFound(shortUrl, ErrorCode.URL_INFO_NOT_FOUND)));
}

I get the shortUrl from the request and the magic begins. First I go to the cacheService to see if the UrlInfo is already there, if not, the Supplier given is called. Since the reactive approach of Redis cache in Spring is not supported by @Cacheable and its ease, as far as I know, I decided to use this approach. Maybe I could use AOP, but I thought that this way matched well with the rest of the code, full of lambda functions, or Functional Interfaces, as it is described in Java. Then I call a doOnNext() just for logging purposes, and then a flatMap redirecting. If empty, the application responds with a 404 status. The method that generates a short url works similarly in the reactive sense although different in logic, you can take a look in the code at the repository.

The implementation of CacheService is interesting because it leverages Reactor's features exclusive for caching:

public <T> Mono<T> getFromCacheOrSupplier(String key, Class<T> clazz, Supplier<Mono<T>> orElseGet) {
  return CacheMono.lookup(k -> redisOps.opsForValue().get(buildKey(key, clazz))
                  .map(w -> Signal.next(clazz.cast(w))), key)
          .onCacheMissResume(orElseGet)
          .andWriteWith((k, v) -> Mono.fromRunnable(() ->
                  redisOps.opsForValue().setIfAbsent(buildKey(k, clazz), v.get()).subscribe()));
}

Using CacheMono I can get reactive control flow exclusive for caching, it also made possible the usage of my supplier in such a smooth way using onCacheMissResume. But first it uses the lookup method to call Redis with the given key, very straightforward since Redis reactive interfaces have the same usage as the blocking ones, but delivering a Mono<T> to us. Finally, in case of miss, the value is cached using andWriteWith. In any way, the object is returned to us, continuing the flow. Very interesting logic and please tell me if there is anything to improve!

Despite not getting into details about the POST route, it is interesting to see the implementation of saveIfNotExists from the UrlInfoService, which is called by the handler to generate the short url and save, but only if it does not exist, the chain of three methods is called:

// Tries to find the long url, if empty, save the request
public Mono<UrlInfo> saveIfNotExist(UrlGenerateRequest request) {
  return findByLongUrl(request.getLongUrl()).switchIfEmpty(save(request));
}

// This is where hashids comes into play, encoding the id given by url range
private Mono<UrlInfo> save(UrlGenerateRequest request) {
  Integer id = getUrlId();
  String hash = hashids.encode(id);
  return urlInfoRepository.save(
          UrlInfo.builder()
                  .longUrl(request.getLongUrl())
                  .shortUrl(hash)
                  .expiryAt(request.getExpiryAt())
                  .build()
  );
}

// Get the current value of the range and deals with range shortage as well
private Integer getUrlId() {
  if (!urlIdRange.hasNext()) {
      Integer counter = sharedConfigurationService.getSharedCounter(urlRangeKey);
      urlIdRange.calculateRange(counter);
  }
  return urlIdRange.getCurrentValue();
}

The last important part that is left to be mentioned is the data layer using R2DBC. This is also very straightforward since Spring Data R2DBC repositories doesn't differ much from the standard, differing only by returning Mono or Flux. One thing I noticed though is that I didn't find a way to map table relationships on R2DBC, there is no javax annotations or Hibernate annotations, of course, for that stuff and I didn't find any reference to that on the documentation. However, I may be mistaken. Let me know if you know anything about that.

Conclusion

It was a really enjoyable experience to get a diagram of an application architecture, add your own ideas and materialize it in code. I hope I can find more articles of this type in the future. There was other experiments I didn't mention here such as documentation and testing, take a look at the repository to see what I did about that, I will probably write another post about it. Thank you for all who reached the end of the post (it was a long one) and let me hear what you think!

A love letter to TypeScript

Leonardo Teteo — Wed, 07 Aug 2019 02:08:43 +0000

Dear TypeScript, I know that your brother JavaScript sometimes may not want to see us together, but I think your qualities are worth it, that's why I love you!

Jokes apart, this is really a post that I have been desiring to write for a long time since I started to actually study the many features that TypeScript have and how wonderful it is. The code in the cover image is not just for show, in a simple example like that I can show many reasons why I'm in love with TypeScript and I want to use it as much as possible from now on. Therefore, without further ado, let's go write this love letter bit by bit.

When we talk about JavaScript we always think about flexibility. It confers a lot of flexibility in the way you write code, JavaScript is very permissive, you can add fields in objects, remove fields, functions can be passed as parameters, arrays can hold elements of any "type" (JavaScript has a very loose concept of type). However, all this flexibility has a cost. When I see a application with more than 100 lines and 1 file written in JavaScript it always looks a mess to me, no offense. I don't know from where things comes from and where they go. It is like stepping into a minefield. My background is from Java, where you have control of almost everything you code, way too much control sometimes.

That's where TypeScript comes to play, it mixes the flexibility of JavaScript, but putting it at bay with the control that static typed languages offer. For me, it have the best of two worlds: JavaScript and Java (never confuse them). So, let's go to the example. First, the Letter interface.

export interface Letter {
  from: string;
  to: string;
  message: string;
}

Okay, pretty normal, right? One thing that I like in interfaces in TypeScript is that you can have both properties and methods defined and the class that will implement them will need to have all these properties to satisfy it. It is different in Java, where "properties" are in the end constants inside an interface, they are not properties. Interfaces in TypeScript, however, show their power along with other classes. So, we present below the LoveLetter class.

export class LoveLetter {
  constructor(public from: string, public to: string) {}

  get message(): string {
    return `I love you, ${this.to}`;
  }
}

Just by looking at the class you may not realize it if you don't know TypeScript wonderfulness, but this class completely satisfies the Letter interface, 100%! First, the constructor. In Java it is always annoying to have to add parameters and assign them to the class properties right after, it is unnecessary boilerplate! I love TypeScript approach where by inserting public, private, etc. to the constructor's parameters you are declaring a class property and assigning the parameters' value to it all in one go. Therefore, LoveLetter class has the string properties to and from that Letter requires.

"But how about message? In LoveLetter it is a method, not a string property!", you might say. That's another very interesting feature of TypeScript: "getter methods". It can be used much like getters in Java, for example, but I think in TypeScript it has a gotcha that Java doesn't have. When you add the get word before the function, TypeScript will treat it as if it were a property! So, we can do this: letter.message instead of this: letter.message(). Since it is considered a property to TypeScript it is a okay to satisfy the Letter interface. It is very useful when you want to shape the data in another way, for example, for visual purposes, like I did with "I love you, #name#", using string interpolation to insert the name of recipient. It may not look a big difference in a simple example, but when you are trying to satisfy some big interfaces or having more complex use cases, it comes in handy.

For some who may be used to Java it may sound strange that the class implements the interface like that without any formal declaration like class Foo implements Bar in Java, but it is one of the reasons why interfaces are so powerful in TypeScript, the class satisfies the interface merely by its shape, a formal declaration is not necessary. You can declare formally, however:

export class LoveLetter implements Letter {
  constructor(public from: string, public to: string) {}

  get message(): string {
    return `I love you, ${this.to}`;
  }
}

The declaration is exactly like Java and that way you will have error messages at the class level if the class is not satisfying the interface, but generally you will not have to be that explicit. So, if one class satisfies an interface only by shape, then you can go to the last part of our example:

const sendLetter = (letter: Letter) => {
  console.log(
    `From: ${letter.from}
    To: ${letter.to}

    ${letter.message}
    `
  );
};

const loveLetter = new LoveLetter('Developer', 'TypeScript');
sendLetter(loveLetter);

Here we have a function that is assigned to a constant and is declared as an arrow function. In this case it doesn't have any special meaning, but arrow functions can make the code more elegant, principally when TypeScript/JavaScript are the home of callbacks. Also, it can solve some concerns around scopes, principally when it comes to this, but we can talk about that in the future. Then we create a new LoveLetter and assign it to a constant, nothing special, but the next line show some of the power of interfaces. Although the function receives a Letter as parameter, since LoveLetter satisfies the interface 100% then you are good to call the function with an instance of LoveLetter as well! And what it is better is that you don't need to tell TypeScript anywhere that, it knows it automatically. Again, the interface is evaluated by its shape, if the shape is right, you are good to go!

It has been very interesting to learn more about TypeScript everyday and how it fuses the things that I like the most in the languages I already know in one beautiful language. There are many other features that are worth mentioning, but I think with this simple example you can get a sense of TypeScript greatness. I believe that TypeScript has a bright future in the market and I recommend it to everybody. Don't worry, I will not get jealous by the popularity of my dear TypeScript.

Is it a good idea to use CLIs to generate projects in JavaScript?

Leonardo Teteo — Wed, 19 Jun 2019 12:21:01 +0000

When I generate a project using Angular CLI, React Native CLI or even once with VueJs it is always somewhat troublesome because sometimes they ask questions before generating the project that as a beginner, you don't understand half of what they are asking and when it is generated, there are many files you don't know what they do.

I am not a total beginner in Javascript, used jQuery for years and even Angular 2+ in some personal projects. I certainly know more about Angular fundamentals like components, modules, services, dependency injection, two-way binding and so on than what all that configuration files even mean.

Is it everything thing there necessary at the beginning of a project? For a true beginner who want to start quickly, it is not one more wall they need to overcome?

The magic known as MySQL FULLTEXT index.

Leonardo Teteo — Tue, 11 Jun 2019 21:03:19 +0000

We developers, most of the time, are having a battle with databases about performance. It is not rare that the root of a performance problem is ingrained deep inside your query. It is a fate we must fight against everyday. This week I had to fight against it and I had some interesting results and new experiences to talk about. That was when I discovered the magic called MySQL FULLTEXT index.

The problem

When you work in a small company we end up venturing yourself in fields that it is not necessarily your specialty, that's me as a developer dealing with databases. I'm not a DBA and for that reason I don't know all the intricacies of a database like MySQL. When a problem comes up everybody helps giving their suggestions, researching and etc. This time the problem was about performance in a search feature of an application.

Giving a similar example of the use case, we have a search feature that looks for products. As expected, there are many fields in a product where you need to search to make it as accurate as possible, some of them are the name of the product, its description, info fields, some keywords, the name of the manufacturer, etc. Only with this description we can have three tables:

Products, self-explanatory
Keywords, it has a separate table because we can reuse for multiple products.
Manufacturer, also self-explanatory.

However, the problem here was not the number of joins (there are many, more than the ones I describe in the simpler example above), but the way the fields were searched for a given search input. Initially we were using a function on the database that made use of REGEXP to look for rows that contained the search input entirely, i.e, if the input was "discount" the column must have the entire word "discount" somewhere to be brought by the database. Okay, looks inoffensive, until the moment it is not.

I still don't know why, I think it would be a good reason to research about and make another post another time (if you have any idea why let me know!), but after divide every part of the query and test all of them we reached the conclusion that the use of this search method was slowing down the query enormously, like, they query was taking many seconds in a production environment, very bad news.

To give some more context so that it doesn't look like it was a big mistake from our part, the system is being rebuilt, it has been running for many years and now we are reshaping the layout and practically rewriting the backend with new technologies and solutions, but we are still using the same database and many queries were not touched yet. Besides, the problem appeared recently, maybe because of the increased traffic. During tests and early months of production it was not detected, the search was performing reasonably well although we had plans to speed it up even more.

The solution

Solve the problem was not that easy due to our time and budget limitations, it would be impossible to do something outside of what we already had of infrastructure, for example, add another database just for that: we thought about Elasticsearch or even any other NoSQL database like MongoDB, some tests indicated that it would give a good performance. However, add another database to the mix added complexity, not to mention cost. Since we are rebuilding we are willing to try new technologies, but we have added Redis for caching purposes and add another database would raise the costs too much.

With this limitation in mind the only thing we could was trying to make the query better in some way. The first thing we tried was to replace the REGEXP for LIKE and although the performance improved it didn't bring the search results we were expecting. At the time I didn't know FULLTEXT index existed and didn't think that MySQL had such feature, I always relate full-text search with Elasticsearch and other NoSQL databases which has this as main feature. It was when I looked for "mysql fulltext" on friend Google and it came with interesting results.

Like any other index you start by creating it, for example:

CREATE FULLTEXT INDEX product_search_index ON product(name, description)

And then the magic begins. To use the capabilities of the FULL TEXT INDEX you can use the MATCH ... AGAINST ... function, there are some modes and you choose the one best suited for your needs. I'm still studying myself about all the modes and their advantages and disadvantages, but you can find all the explanation in the documentation, for now I will stick with IN NATURAL LANGUAGE MODE, the default one and the one that best suited for my use case, coincidentally. Quoting the documentation this mode works like this:

A natural language search interprets the search string as a phrase in natural human language (a phrase in free text). There are no special operators, with the exception of double quote (") characters. The stopword list applies.

It looks for the words we are looking for as a phrase in free text. MySQL has a stopwords list that can be configured for your needs to improve the results. Then you can make a simple query to bring the results:

SELECT * FROM product WHERE MATCH(name, description) AGAINST ('notebook off discount')

The query, then, will bring the results of the search in order of relevance. Furthermore, it is fast, really fast comparing to what we have before. We brought down the query response time from many seconds to less than 1 second most of the time. Everybody was surprised, including me, about the results. It was like magic and this makes me think I need to study this more to understand how they do this.

Conclusion

It is impressive like everyday you can learn something new and sometimes it can turn out to be surprisingly the solution you were looking for. With that I was able to solve the problem with only the resources I already had and ultimately not overengineering the solution.

Do you make the same project with two or more frameworks/languages just to practice?

Leonardo Teteo — Mon, 10 Jun 2019 13:59:43 +0000

When I come up with a project idea I always have this same dilemma. I want to use new technologies, but also to continue improving in what I know. Should I make the same project many times with different languages just to practice?

The Tools and Skills Needed to Tackle Microservices

Leonardo Teteo — Wed, 05 Jun 2019 03:18:01 +0000

It is almost 2 years since I started to develop professionally and really understand the challenges that it comes with building and maintaining services up and running 24/7 without breaks. Since then, I became fascinated to the concept of microservices. It looks so simple, the old divide and conquer, but brilliant. However, it comes with a lot of challenges as well and many tools to tackle these challenges. That's where my question is born. What are the tools you use to build your microservices architecture?

It seems a question that I could easily Google it and get some answers. I did it sometimes, even listening to other companies infrastructure. A lot of them talk about Kubernetes as a mean to deploy microservices. Others deal with communication, which seems a real challenge in microservices, like Kafka and other messaging or data streaming tools. Service discovery and circuit breaker are some names I read as well. On the Java world, where I live most of my time, there are Netflix OSS with Eureka, Feign, and others. It is a myriad of tools and begs the question about what do you use and your stories with those tools.

Furthermore, all stories we hear about microservices deployments is from big companies where the infrastructure is massive and it sounds impossible to practice with those tools unless you are already in a company with such a massive infrastructure. Am I wrong? How do you practice with those tools to understand them well?

I understand that most of the tools used depends on the user case, but that's why I asking here! I hope we can gather a lot of different stories of all shapes and sizes! So, let the talks begin! :D

What do you do to practice new programming languages and/or frameworks?

Leonardo Teteo — Sun, 20 Jan 2019 13:34:13 +0000

I've been struggling with this problem for a while now. I'm curious, I like to experiment new things, be it languages or frameworks, but for that it is important to have some idea to code with. Just do the tutorials is not enough for me, it is generally too easy, so I need something more real to practice. However, I don't have much creativity to projects. I try to think about some problem that I may have and would like to automate or something linked with my hobbies that would be nice to create. I do have some projects slowly evolving, but since they are a little more complex, I tend to use things I'm already fairly familiar with, so it doesn't count. I also try to find open source projects on GitHub that I could contribute, but the well kept ones are generally too hard for someone who wants to practice the basics, it is intimidating.

So, the question is, what to do when you want to practice something, but doesn't have an idea of project to try it on? There are sites that can help with that? Front-end, back-end or both. What do you do when you are in this kind of situation?

Building Twitter layout with Flutter

Leonardo Teteo — Sun, 13 Jan 2019 18:30:34 +0000

Flutter is the new shiny framework for mobile development by Google, it was first released last year and this month it reached the 1.0 version. I've known Flutter for some months now, but since it reached 1.0 I decided to take a deeper look at it. I've developed mobile applications before using native Android with Java and cross-platform with Xamarim and NativeScript, it was not anything serious, but it is an area I find interesting and some people can argue, it is the near future.

Presentations done, now let's go to the juicy details. In my opinion, the most remarkable feature of Flutter is how easy it is to understand its layouts mechanics. Basically, everything is a widget and you can put a widget anywhere you want. The widgets are very descriptive, have attributes that cover almost all common usages, it is effortless and simple to build a layout. Since everything is a widget, the framework is very permissive, dynamic and logical, you have no fear of doing something wrong, just do it, and with hot reload as fast a blink you can test everything easily.

My experience with Flutter is amateur level, I did the most basic tutorial on Flutter website, read some things about how Flutter handles layouts, principally this great article that summarizes everything you need to know about Flutter layout structure. I would never dare to try replicate Twitter layout in none of the aforementioned frameworks, but with Flutter I had enough confidence to do it because I had no fear of failure.

So we have something like this:

To make it simpler, I made the layout taking in consideration only simple tweets, meaning only with text, no images, quotes, etc. Just to start. However, the rest will be present, the app bar at the top, the floating button to tweet and the bottom bar with tabs. Does it look too complex? Divide and conquer is the best approach for that!

First, let's see the app bar at the top. We can see that we have a circle avatar and a text written "Home". An app bar is a very common design pattern in Android apps and Flutter knows that,because of that Flutter has widgets to make it easy to build one. To do that we use the Scaffold widget that has a parameter called appBar that accepts an AppBar widget as follows:



return MaterialApp(
      title: "Twitter Layout",
      theme: new ThemeData(
        primaryColor: Color.fromRGBO(21, 32, 43, 1.0),
        accentColor: Colors.blueAccent[200],
      ),
      home: new Scaffold(
          appBar: new AppBar(
            title: Row(
              children: <Widget>[
                Container(
                  child: CircleAvatar(
                    child: new Text("L"),
                    radius: 15.0,
                  ),
                  margin: EdgeInsets.only(right: 30.0),
                ),
                new Text("Home")
              ],
            ),
            elevation: 4.0,
          ),
          body: _buildBody(context)
      ),
    );

As you can see, the Scaffold widget is wrapped by a MaterialApp widget that makes is it easy to build apps using Material Design, design directives for Android apps by Google, which the Twitter app clearly uses. The AppBar widgets has a title parameter that accepts any widget, although generally it is just a Text widget. Therefore, I created a Row widget that is responsible to place its children aligned horizontally, the way we want in this case. We add the children of the Row widget in a children parameter. In this case the widget has a Container and Text as children. The Container widget is a very convenient widget because it enables you to custom spacing, positioning and painting configurations to its child. In this case, I added a CircleAvatar widget as its child and added a margin of 30 pixels to the right of the circle to give separation from the "Home" text. The child of a CircleAvatar widget can be any widget, but we generally use text or an image, to make it simple I used a Text with the letter "L". The radius indicates the size of the circle. If you notice, the circle on the app bar is a little smaller than the ones on the tweets, so I had to configure that. The parameter on the AppBar widget indicates the elevation of the app bar, it is what gives that shadow below it. In the end it was not necessary to make an app bar, a circular avatar and all the spacing configuration from the ground up, Flutter makes it easy to achieve the result you want with the use of common patterns. The Scaffold widget also has a body parameter, where, in this case, all the app content resides. Let's get into it.

To organize the layout I made the body building logic on a _buildBody() method which is as follows:



Widget _buildBody(context){
    return new Container(
      child: new Column(
        children: <Widget>[
          new Flexible(
              child: Scaffold(
                body: new ListView.builder(
                  itemBuilder: (_, int index) => _tweets[index],
                  itemCount: _tweets.length,
                  reverse: false,
                ),
                floatingActionButton: FloatingActionButton(
                    onPressed: null, child: Icon(Icons.edit)),
              )),
          new Container(
            decoration:
            new BoxDecoration(color: Theme.of(context).cardColor),
            child: _buildTabsBar(context),
          ),
        ],
      ),
    );
  }

The _buildBody() return a Widget, which is, in this case, a Container. The container has a Column widget as child, which all the magic happens. In the image used as base above we can see that we have the list of tweets and right below it the tabs bar, besides the floating button. The children of the Column widget are placed the way they appear on the screen. First, we have a Flexible widget and, as the names suggests, it is flexible, meaning in this case that it will occupy the remaining of the space of the parent, this is where our list and floating button are. The Flexible widget child is a Scaffold widget. Nobody said that it can be used only was the main widget of a an app like we used before although it is very common that way. It is used here to facilitate the creation of the floating button since it has a floatingActionButton attribute, which takes a FloatingActionButton widget with an edit icon as child. It also has an onPressed attribute to configure the event, but since the goal here is just to make the layout, it is null for now. Only this will make the floating button appear exactly where we want it. As before, on the body parameter of he Scaffold widget, we have the main content, the list of tweets, that is implemented using ListView, which is kind of a specialized column that enables scrolling when the content does not fit entirely in the widget. It simply builds the list from a list tweets, which we will see later.

For now, let's focus on the other child of the Column, the container which has the method _buildTabsBar() as child. Let's see how the tabs are structured.



Widget _buildTabsBar(context) {
    return Container(
      height: 60,
      color: Color.fromRGBO(21, 32, 43, 1.0),
      child: new Row(
        mainAxisAlignment: MainAxisAlignment.spaceEvenly,
        children: <Widget>[
          Icon(Icons.home, color: Theme.of(context).accentColor),
          Icon(Icons.search, color: Colors.grey[100]),
          Icon(Icons.notifications_none, color: Colors.grey[100]),
          Icon(Icons.mail_outline, color: Colors.grey[100])
        ],
      ),
    );
  }

Making the bar is very simple and that's what makes it fantastic. It is just a Container with a Row as child that has the icons as children, that's it. I configure the height of the container and its color. Then, in the Row I used the mainAxisAlignment attribute to make the space be evenly distributed between the icons making look like the original layout. I still remember how it is a pain to center things using HTML and CSS, not to mention make the space even.

So, let's see the last part, the tweet. It looks like the part where there are more elements to take care of and it is right, but it is not that much more complex than the rest of the layout if you visualize it as a collection of rows and columns. To make the tweet reusable and easy to use in a ListView I created a Tweet class containing all the layout for the tweet:



class Tweet extends StatelessWidget {

  Tweet({this.user, this.userHandle, this.text});

  final String user;
  final String userHandle;
  final String text;

  @override
  Widget build(BuildContext context) {
    return Container(
      padding: EdgeInsets.all(10.0),
      decoration: BoxDecoration(
          color: Color.fromRGBO(21, 32, 43, 1.0),
          border: Border(bottom: BorderSide())),
      child: new Row(
        mainAxisAlignment: MainAxisAlignment.start,
        crossAxisAlignment: CrossAxisAlignment.start,
        children: <Widget>[
          CircleAvatar(
            child: Text(user.substring(0, 1)),
          ),
          _tweetContent()
        ],
      ),
    );
  }

...

}

To make it simple, it is stateless and only gets the user, user handle and the tweet text to build the tweet. A full-fledged tweet would probably be statetul because of the counts for likes, retweets and answers and have much more parameters. The build() method returns a Container with the basic layout configuration, padding and a border at the bottom acting as a divider for the tweets, then it has a Row as child. The row contains the circle avatar of the user who tweeted and the tweet content, which is built through a method for organization sake. mainAxisAlignment and crossAxisAlignment set as start is very important because it makes the circle avatar to be placed at top-left corner as needed.

The _tweetContent() method returns the content of the tweet, of course. Below is the entire method since it is easier to connect the dots by seeing all elements, after that I will explain the code.



Widget _tweetContent(){
    return Flexible(
      child: Container(
        margin: EdgeInsets.only(left: 10.0),
        child: Column(
          crossAxisAlignment: CrossAxisAlignment.start,
          mainAxisAlignment: MainAxisAlignment.start,
          children: <Widget>[
            Row(
              children: <Widget>[
                Text(user,
                    style: TextStyle(
                        color: Colors.white,
                        fontWeight: FontWeight.bold)),
                Container(
                  margin: EdgeInsets.only(left: 5.0),
                  child: Text(userHandle + " · 30m",
                      style: TextStyle(color: Colors.grey[400])),
                )
              ],
            ),
            Container(
                margin: EdgeInsets.only(top: 5.0),
                child: Text(text, style: TextStyle(color: Colors.white))),
            Container(
              margin: EdgeInsets.only(top: 10.0),
              child: Row(
                mainAxisAlignment: MainAxisAlignment.spaceBetween,
                children: <Widget>[
                  Row(
                    children: <Widget>[
                      Icon(Icons.message, color: Colors.white),
                      Container(
                        margin: EdgeInsets.only(left: 3.0),
                        child: Text("15",
                            style: TextStyle(color: Colors.white)),
                      )
                    ],
                  ),
                  Row(
                    children: <Widget>[
                      Icon(Icons.repeat, color: Colors.white),
                      Container(
                        margin: EdgeInsets.only(left: 3.0),
                        child: Text("15",
                            style: TextStyle(color: Colors.white)),
                      )
                    ],
                  ),
                  Row(
                    children: <Widget>[
                      Icon(Icons.favorite_border, color: Colors.white),
                      Container(
                        margin: EdgeInsets.only(left: 3.0),
                        child: Text("15",
                            style: TextStyle(color: Colors.white)),
                      )
                    ],
                  ),
                  Icon(Icons.share, color: Colors.white)
                ],
              ),
            )
          ],
        ),
      ),
    );
  }

The parent element is a Flexible widget which allows the element to occupy the rest of the space available, the space on the row created on the build method above. Then we have a Container which allow to configure the margin necessary on the left of the content to separate it from the circle avatar. Finally, we have a Column, where the fun starts.

Again we have crossAxisAlignment and mainAxisAlignment as start to make the content to be positioned a the top-left corner. The first child of the column is a Row that contains the user name, its handle and the time of the tweet. To make the user handle be slightly separated from the user name we wrap the Text into a Container as we've done before. The second child of the column is a Container that wraps the actual text of the tweet with a margin at the top. Finally, we have another container with a Row as child to build the reply, retweet, like and share buttons. mainAxisAlignment is set as spaceBetween so to that all icons will have an equal space between them and will be evenly distributed through the width of the row. Very convenient configuration, positioning and spacing was always something very troublesome for me in other frameworks. Each icon is a Row itself containing the icon and the number of replies, RTs, etc beside it. Simple, isn't it?

So let's see the final result below:

It is close enough except for the icons which are probably Twitter exclusive. The tweets are just a sample.

I hope this post will be of any use for people who are thinking about taking a look at Flutter and how to get started. For me it was love at the first sight when it comes to layout creation and unless it is limited about interactivity, where I still have to experiment, I believe that it will be popular very soon!

Is it good or bad to be all over the place with dev technologies?

Leonardo Teteo — Mon, 24 Dec 2018 00:15:36 +0000

2018 has been a wild year for me, it was the first year of my professional career as developer, before that I dealt more with networking infrastructure and administration. How I went from a career towards SysAdmin to Developer is another story for another day. Before starting really working with development I've never dived deep into the development world, I was still at college studying to get graduated, there wasn't much space, specially because it was not my first choice at first. But I was caught by the development world and here I am, one year ago, writing a retrospective and somewhat asking for advice about my eagerness to learn a lot of things at the same time.

Let's start from the beginning, I was hired for a Java Web Developer position, and, I have to confess that days before the interview I had no knowledge of Spring Framework at all, I mean it, I'm not ashamed for saying the truth. I knew Java from college and has developed simple web applications in the past, but it was in C# despite the MVC concepts are the same. That was what helped me in understanding Spring, JAX-RS and other frameworks from Java quickly. Fortunately, I was hired and I started to study Spring deeply while at work and outside of the company. Spring Boot, Spring Security, Spring Data, Spring Session, etc. I still have a lot to learn though.

In the meantime, the company is also a AWS partner, we usually deploy our applications on AWS and the knowledge of its services is paramount. So, I also started to study AWS with the goal of getting certified. After some months of study, I was able to pass the certification exam and I'm now a certified developer. It helped me a lot in having confidence to propose improvements to what we already had on AWS and it has been great to see this come to fruition.

Only that is already a really good amount of knowledge to swallow, Spring and AWS have some things in common: they are huge, with a lot of modules, services, complexities and details, they range from simple web applications to security, data handling, etc. I also put some effort on understanding DevOps on AWS, which was not really part of the exam that much. I also took a liking for serverless and for that reason spent a long time making experiments with AWS Lambda and DynamoDB.

Since I was doing experiments with DynamoDB, I also took a look at MongoDB to see what differences they had, I saw that DynamoDB had some limitations during my experiments and I had to take a look at other solutions to see if they were more compatible with my user case. Right now I'm experimenting with Elasticsearch and Logstash and enjoying a lot as well, it can have a very good impact in future projects in my opinion.

However, before I enter into the current Elasticsearch vibe I also had the front-end vibe, I was worried about my knowledge of HTML, CSS and frameworks such as Angular, VueJs and React, I also made a post about Full-stack development, I was and I am still worried about how to become a full-stack developer. My first adventures with Angular was not that pleasant, but recently, after so much time making web applications, many of the experiences I had with Spring and template engines such as JSP and Thymeleaf, proved to be very useful in understanding Angular better, right now I going back to Angular and developing an UI for a personal project, maybe soon I will talk about that here.

Besides all that, I'm also right now experimenting with Flutter, the newest and shiny framework for mobile development by Google. This year I also played around with native Android development with Java, NativeScript and Xamarim! Flutter has been a pleasant surprise and I think this vibe will last through next year.

So, after all that I think it is possible to understand what my worries are: I'm all over the place! I experiment with a lot of things from databases, mobile and web frameworks, cloud providers and so on, without going deeper into any of them, excluding Spring and AWS, which actually makes part of my daily professional life.

What are your takes about this? Should I focus more? Is it all right to experiment so much without going deeper? What are your experiences on dealing with the huge amount of technologies we have at our disposal nowadays and the curiosity IT guys like us naturally have?

Happy holidays! :)

The node_modules problem

Leonardo Teteo — Mon, 19 Nov 2018 22:45:21 +0000

I think I'm not the first one to talk about this problem even here in dev.to. I quick research trying to found any solution concluded with the image that is the head of this text. The node_modules folder is where your project dependencies are stored, common knowledge. Its weight is also common knowledge.

Why I decided to vent my frustration now

Black Friday is here! It means discounts and the opportunity to update your computer. Therefore I decided to buy a SSD to boost the performance of my laptop, from 1 TB HDD to 500 GB SSD. All my files right now sums 299 GB, so I will not lose much space, but I decided to do the housekeeping work even so, this includes making backups of my projects. Not all projects I make I put on GitHub, sometimes I'm just experimenting and it is not worth the trouble, but I keep them anyway.

When I started the copy & paste process I remembered how heavy node_modules are...

Some comparisons

One example that shows clearly the problem is the node_modules folder of my ToRead CLI project as you can see in the imagem below.

The size of the folder is not really the problem although I will get to that later, but 15.000 files and more than 1800 folders!? Are you kidding me?! It is a simple CLI project with 5 files! Just for a comparison, let's see how many files and folders there is in the Windows folder:

While the system was counting I really thought node_modules would win this, but no. In any case, the folder has almost half of the amount of files an entire operating system has!

As I've said, the problem when copying node_modules folder from one place to another is not the size, it is the amount of files and folders, the complexity of the tree. It is a nightmare for a HDD. It takes many minutes to discover all files let alone copy them. In the end, it also impacts npm performance and there are memes for that also.

Other comparisons come from my passion for serverless. It is not rare for me to implement the same function in both Java and Javascript and since you have to bundle the function along with its dependencies it is a good way to compare which one is more efficient in dependency management. In one of my projects I wtote the function in both languages with virtually the same features and Java bundle size is 11.1 MB and NodeJS bundle size was 29.0 MB. Therefore, NodeJS can do a better job at the size of dependencies as well.

What other languages do

Besides NodeJS I have experience dealing with dependencies in two more languages: Java and C#. They have, in my opinion, a very similar way of handling dependencies and a much more efficient way than NodeJS.

Java has Maven, Gradle and other dependency management applications that works basically the same. There is a remote repository of the dependencies, generally Maven Central and a local repository. Maven always looks for the dependency in the local repository first and if not found it downloads from the remote repository. The dependencies are not within the project, like node_modules folder, it is more global, it is downloaded once and can be used by many projects, just add to your pom.xml.

C# follows the same idea, you list your dependencies in a .csproj file and Nuget takes care of the dependencies having also a remote and a local repository. It is much more efficient to handle dependencies this way, download once use in any project locally.

I think there is also a difference in culture and the way the languages were structured and what people see as libraries. Java has a very mature core of libraries that can deal with almost anything, common scenarios or not. Therefore, libraries in Java generally are meant to be an abstraction of what Java already has, making it easier to use. Therefore, the libraries have a more shallow dependency tree, reaching the Java core libraries much quicker.

What I see in NodeJS on the other hand is the opposite, everything can become a library, even a library that sums two numbers (hypothetical example, I hope) and libraries are heavily dependent of one another generating deep dependency trees, many files and folders.

Conclusion & Discussion

I am certainly not qualified to criticize NodeJs structure and engineering, but as a user I clearly see a problem and some lessons from other languages that can be used to improve the dependency management, which is paramount nowadays for almost every application. What do you think this problem came to be and what has been done to solve it? It would be very interesting to hear from more experienced developers what you do to remedy this.