DEV Community: yellow1912

On the topic of death

yellow1912 — Wed, 23 Dec 2020 05:48:41 +0000

Recently, I noticed that I have more grey hair than I expected, and then I realized I'm turning old. I have this mid-life crisis of life's goals and values. I haven't reached most of the goals I set out, and the thought of missing out on new things that will happen in the far future saddens me. You know, the space travel, the exploration of new horizons, the meetings with aliens perhaps,... I will never ever see that.

I'm not a religious person. I have always been pondering what will happen after death. Will it be the end of everything, or will there be some continuation after death? Whatever it will be, I'm pretty sure that the "me" that exists now will cease its existence completely. Death is a fascinating topic and I have read and watched lots of documents on it.

It's not all grim, however. When I look at my children, still of a very young age, I see a glimpse of hope. Through them, I see a continuation of life. Some day, my grand grand daughter may ride her own spaceship out of the Solar system during her day trip. Some day, we the human may even find the fountain of youth that death seems like a strange concept to us all. Imagine the possibilities when you can live for centuries. Oh the things you can learn, the places you can see, the stuffs you can do.

Using an iphone in 3rd world country can hurt you

yellow1912 — Sat, 28 Nov 2020 06:42:02 +0000

Recently, I was doing speed test on a new website for a client, and I was puzzled by Chrome's report saying that my images were too big.

I looked at the details and it seems like the browser is using a way bigger image than what I think it should use. All my images use srcset with w descriptor to provide different sizes of images for different device resolution, and on iphone it shows an image of 2600px being used. dang. How can it be so big when the phone screen is relatively small? I spent hours thinking it was a bug and tried various ways to fix it.

It turned out that new devices have this thing called DPR (device pixel ratio), and apparently high end devices may have very high DPR such as 3 or even more. What it means is that now if the normal css pixel is 500, you need an image of 1500px for the given device.

This can have huge impact on the site performance because the device needs to download 3 times more data (more or less). For third world countries like mine where 4g feels sluggish, this can really delay the rendering of the site. For the clients they have to wait longer, and also unknowingly pay for more bandwidth.

Do we really need those big images? Only sometimes. And should not come at the cost of potential users abandon the site due to slow load.

For people using lazysizes, you can use the optimumux plugin to set the optimal DPR. The default is 1.6 mx which is reasonable IMHO.

For above the fold content which should not be loaded lazily IMHO, I'm looking into the setting of srcset with the x descriptor and sizes. I will update this post once I have a solid answer.

PHP 8 has finally arrived

yellow1912 — Thu, 26 Nov 2020 18:02:37 +0000

PHP 8 is now available for download. Hate it or love it, PHP has come a long way. I cannot wait to finally bump my PHP version on dev server and see how things go (fingers crossed).

The PHP team has done an amazing job. Thanks guys.

A super weird case with SVG performance

yellow1912 — Tue, 24 Nov 2020 09:40:32 +0000

(Image from By W3C, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=89557878)

I have this super weird case where I need to display a big tree of text with svg icons. For some reason on the latest Chrome it took almost 30seconds just to render the whole tree (insane). I took a look at Webflow which has a similar tree and there seems to be no issue at all.

My tree was rendered by Vuejs 2.x, I'm not sure if there is anything to do with that, I could not figure out why. The only thing that I could think of, is that inside the Vuejs template I use something like this to render the svg:

<svg>
  <use xlink:href="#general_small--arrow_2_down"></use>
</svg>

(The svg sprite is printed on the top of the body tag with display none)

In any case, I put in this temporary fix which manages to bring down the rendering from 30s to 2s:

Add a class to hide (display:none) all svg inside the tree every time I need to re-render the tree
Set time out (something like 200ms)
Remove the class

And now all the SVG icons are shown nicely, instantly.

Edit 1:

So I think I found the issue. The inline sprite svg file that I was including was huge (hundreds of svg files), and it seems like that slows down the lookup drastically (even though the file was already included inline in html). By splitting the file into smaller files I was able to speed up the rendering multiple times.

Create a log service with PHP and Clickhouse [Part 2]

yellow1912 — Sun, 11 Oct 2020 10:56:50 +0000

With Vector taking care of mining and shipping our logs, we need a place to store all these logs.

I built this logging service for simple and quick debugging and alerting for dev team. My goals are: fast and cheap. The log service can fail and it's okie to fail and get restarted again quickly.

I came up with the following table schema for my log (I want it to be as generic as possible to store logs from different sources)

CREATE TABLE IF NOT EXISTS log (
        instance_id String,
        node_id String,
        project_id String,    
        app_name String, 
        app_version String,
        client_ip String,            
        log_level String,
        log_message String,
        log_code Int32,
        log_context String,        
        log_created_timestamp UInt32,
        log_time DateTime,
        log_date Date DEFAULT toDate(log_time)               
    )
    ENGINE = MergeTree()
    PARTITION BY toYYYYMM(log_date)
    TTL log_date + INTERVAL 1 Week
    ORDER BY (project_id, log_created_timestamp)

There are several things I would like to point out:

I don't care about old logs, for now, so I decided that logs older than 1 week can be deleted.
I want logs to always be listed by created date (newest logs first).
I want to filter by certain criteria (to group logs by application, client, etc)
I'm a Clickhouse newbie, and I'm still figuring things out. I secretly hope that someone will look at the table schema and point out improvements that I can do.

In the beginning, I logged error logs as well as access logs, and the number of records quickly reach billions after just a few days. The good thing is that the log service still runs surprisingly well on a rather small server instance (CPU: 2 vCore RAM: 4096 MB Storage: 80 GB SSD). The bad thing is that listing keeps timing out. I later found out that I didn't set up correct indexes for my table (you really really must setup indexes for the columns you want to filter and order by).

Next time I will show how I handle the logs using PHP.

Create a log service with PHP and Clickhouse [Part 1]

yellow1912 — Tue, 29 Sep 2020 15:16:57 +0000

I would like to share my experience configuring a very simple logging service with PHP and Clickhouse. I would like to stress that this is by no mean an optimal setup, I simply want to share my experience and I would love to hear feedbacks to improve.

Requirements

A very simple logging service to track all kinds of events such as nginx error or php error internally and externally. I find ELK too complicated to properly setup and guard, and too resource intensive (and thus costly) for my needs.
Performant enough to handle tens of millions of logs per day on an average server (I pick the Vultr instance below which has 2 CPU and 4 GB of RAM). I'm handling over tens of million logs per day with this setup and still have resources to spare.

Tech stacks

PHP
Clickhouse
Vector

Why PHP?
Because I'm familiar with it and because why not? I don't want to spend days and weeks trying to learn a new thing just for something really quick that I can do for a 1-2 hour with PHP.

Why Clickhouse?
Clickhouse is a very performant, easy-to-setup database designed and optimized for writing data. I have great experience using it to track views and clicks on my website.

Why Vector?
I looked at many log collectors such as syslog, syslog-ng, filebeat etc... I'm not a system guy and their documents are terrible, terrible. Filebeat is not so bad but I was not entirely impressed with it (I used it in the past). Then I found Vector and I was blown away by the website (so clean), the documents, the options, and the support.

Preparation

First, we need to collect the logs. Vector is super simple for collecting logs, you can find the website here with all the documents:

https://vector.dev/

I'm going to share some of my current configs, please do note that I'm a Vector virgin, so please do review before use and please let me know if I can improve anything:

https://gist.github.com/yellow1912/782d53cb97b9de86d4fdb2848cc2e520

You will notice that I'm opting to send the logs to a remote server via http/https protocol. You can choose any other protocol you want. I also pass along an authorization token so my log service can check.

Next time, I will cover Clickhouse database setup.

Please recommend me a tech stack

yellow1912 — Fri, 25 Sep 2020 00:04:00 +0000

This hacktoberfest, I want to give back something to the community where I'm living. I'm thinking of creating a map which you can review the living condition of certain area.

Once you give the lat long you can see reports regarding crime rate, noise, air quality, pollution etc. Each criterion is going to be on a different layer of the map which you can turn on off.
There should also be historical reports to see simple charts and stuffs.
Information may come from official sources or from user reports. Users can also upload reviews and pictures.
The site should be in multiple language.
Scalability would be great.

I code in php and JavaScript and know a bit about golang and lua. I'm also familiar with nginx, redis, mariadb, clickhousedb. I can learn new things if necessary. Thank you ❤️.

I failed the spam report captcha on dev.to (20+ times)

yellow1912 — Tue, 15 Sep 2020 01:06:28 +0000

I saw several spam posts today on dev.to and tried to report. Unfortunately, I failed the captcha like 20 times and finally gave up.

It makes me think though, what is the priority here? We can post without any captcha, but cannot report spam behavior without going through hoops.

On another hand, asking a human to prove himself a human while using a flawed script can be really annoying and frustrating.

Integrate Symfony form with Vuejs

yellow1912 — Sun, 22 Mar 2020 03:55:41 +0000

I think I'm old style because I prefer to separate html from javascript. I don't like how Angular (the new versions) and React force you to mingle html and javascript. Separating html, css, js is just something I've got used to and it's difficult to change that way of thinking.

In any case, I'm working on a huge Symfony (4) based application, and most of my forms were built on Symfony form and twigs. The javascript library I used was Angularjs 1.x and it allowed me to support the "legacy" codebase just fine. I could easily embed Angularjs directive right into the html code by adding attributes/classes.

We are stuck with Angularjs 1.x however, the later version of Angular requires significant changes in the codebase that is impractical for us. That is until we found out about Vuejs. With Vuejs we could still keep our legacy codebase and it was possible to migrate from Angularjs 1.x to Vuejs 2.x without any difficulty except for 1 thing: form.

In Symfony, the form is a very powerful component and can be generated dynamically so it's difficult to know the exact structure of the form (which fields, nested structure, etc). With Vuejs if you want to use v-model you will have to ensure that data is already available.

Let's use an example here (open the console log on codepen to see the error):

You can see that if I have a v-model referring to undefined data I will get an error with Vuejs (in Angularjs it will automatically be set for you).

Many current tutorials on the internet recommend hard-coding the form data structure into your components. This is not good for us because:

Symfony form can have deeply nested structure.
There is no way that we know the form structure beforehand (form has events and transfomers that can take options and modify form's structure).
Even if we do know the form structure beforehand, we will have to create many components for all the forms that we have. This is not practical.

The solution I came up with, was to pre-populate a custom form component with the form data structure. I broke this into 2 phases:

I. Phase 1:

For fast migration, in phase 1 I want to keep all the current twig rendering of the form. For this purpose, I figured out that I could dump the form view (the view data object returned by Symfony) to a prop of the form component (I called it initialData). Using that initialData I populated the data of the form component to make sure that all the fields are pre-populated (meaning that all v-model are referring to valid data).

Note 1: Do note that Symfony form view object is huge, you don't want to dump every single thing in there, just enough for you to build the necessary data structure.

Note 2: Handling prototype was a pain in the a**, but it was possible to do. Prototype in Symfony form refers to the dynamic fields that can be added/removed on the fly. I used Vuejs dynamic components to get around it. Whenever the user clicks on the add button, the form wrapper component automatically adds a child component dynamically that can be rendered by Vuejs (using <component :is="something">).

Note 3: Be careful with Vuejs dynamic components. I had to spend days to debug why part of my form was re-rendered every time I make any change to the form data. It turned out that I was missing keep-alive and I was not "caching" the dynamic components properly.

I don't include any code sample for this here because the dump code that I'm using is rather hacky and only optimized for our app. If you want to see it please feel free to let me know.

II. Phase 2:

In phase 2, I wanted to render the whole Symfony form using Vuejs (so no more twig). The reasons are:

It's faster (Symfony form prototype can generate huge amount of code)
It's easier to re-render part of the form
It just feels better

For this to work, I had to write a helper method in Symfony controller to dump the form view object via ajax request. This was quite simple but also a bit hacky. The form view object contains a huge amount of information, I had to cherry-pick only the necessary information I need.

I also wrote some helper method inside my form component to replicate the functionality of Symfony's form_row, form_widget, ... twig helper methods. These helper methods loop through the block_prefixes (if you are Symfony dev you know what I'm talking about) to pick the exact template to use. These templates are stored as script snippets with the corresponding ids. I believe I could create separate components for each form(input) type but for now, this feels more natural to the team. I could copy whatever code we have on twig and with some minor tweaks turn it into Vuejs compatible code.

I hope this helps someone struggling out there. I know I should share some code but at the moment the code is still unstable, messy and ugly and I feel shameful to share it. If you want though, please do let me know.

Also, if there is a better way to integrate with Symfony form, please do share with me as well. I'm very new to Vuejs.