DEV Community: Amitav Roy

Understanding the secret of secrets

Amitav Roy — Mon, 16 Aug 2021 03:46:17 +0000

Any application that you create will have some secrets. Even a todo application which is storing information into a database has secrets. The credentials that you use to connect to the database is a secret for that application, isn’t it? But, many developers don’t pay enough attention to this part.

I have worked on many projects as a consultant. I would advise people on how to improve their applications. Sometimes, these improvements include code level improvements, architecture changes to infrastructure level changes.

I have seen secrets inside codebase which are actually used in production many times. These things scare me a lot. It's dangerous to commit and push these values inside the codebase.

Dot env files

Most of the modern frameworks allow you keep the environment variables out of github. Laravel for example uses .env files to load up secrets. And, although there is a .env.example file which should be a replica of the actual .env file minus the values. The idea is that if a developer clones a repository, he/she needs to copy the .env.example file and name it .env. And, the values which should go into the .env file. Some common ones will be the app key, database credentials etc.

Configs should have defaults

Laravel allows developers to use config variables which will pick up values from the .env file. But, if that value is not present, it will pick up a default value. For example the default port of MySQL database is 3306. Now, the database.php configuration file already has this. What that means is, if you have not changed the default, you don’t need to define it at all.

It allows the developer to get started quickly and define only the things which are beyond the default configuration.

Private repositories are not enough

If you feel that making a repository private will take care of security, I beg to differ. A private repository means that the code is not available in public. But, it doesn’t mean the secrets are secure. Sometimes, we want to share the codebase with developers. Let's say he will add a new feature. But, do you want them to actually get access to the credentials which are being used on production? Will you be comfortable with the fact that the developer has access to the live email sending service?

Also, you need to understand that deleting the secret code from the code doesn’t mean it no longer exists. A user can always revert to a particular commit and see the keys in history. This basically means, once a key gets committed even by mistake there is no way to get rid of it. You have to start a fresh repo.

Secrets are lying around

Believe it or not, secrets in codebase are a huge problem. Github has scanners which scans your code for such potential problems and warn you as well. Even then, if you go to Github and search for “removed aws keys” and filter the search by commits everything will be clear.

So, I hope I was able to make you understand the gravity of this problem and how you can work towards solving this problem. Most frameworks already have the basic checks in place so that the user doesn’t make some of the common mistakes. However, that doesn’t mean they can stop you from all the other possibilities. And hence, as a developer it’s your responsibility to keep these things in mind.

Do share your experiences on this particular issue. What you have seen, how you have solved the problem, anything. You can find me on twitter or on the Discord Bitfumes community.

Banner photo by Michael Dziedzic on Unsplash

Power of estimation for developers

Amitav Roy — Sun, 08 Aug 2021 15:46:33 +0000

I have been working software developer and a Technical project manager at Focalworks for more than 10 years now. Over the years, I have realised the importance of accurate estimates. Accurate estimates are important for projects to be productive and profitable. It doesn't matter whether it's a fixed price project or a retainer model. Accurate estimates ensure that work is getting delivered on time.

Over the years I have worked on a lot of projects. In some my role was that of a developer and in some projects, I played the role of a project manager. With experience I have realised the importance of estimates. and some things which we should keep in mind while doing that.

In this post, I am sharing some points which you should consider while doing estimates.

Why do estimates?

While estimates can feel like a need more from the non-dev stakeholders of a project. But, it's important for the developers as well. Estimates helps in better timelines and release management.

And, it also helps in gaging failure very early in the project. If the sum of all the estimates is going beyond the actual timeline, you know there is a problem. This helps you decide whether you need more resources or we need to shift the timelines.

Developers will feel more responsible about the timeline when they estimate their work. It's almost like a buy in from them about how much time it should take for them to complete the task.

Tips while estimating work

Although there is no magic rule that you can follow for accurate estimates. There are certain things a developer should keep in mind for better accuracy. And remember, estimation is a skill, the more you do the better you get at it. So, here are a few things which you should keep in mind while doing estimates.

Understand the scope of work

While estimating work, you should have a very clear picture of the work. If the scope is not clear then, as a developer you should do the analysis. Ask questions, and get those answers before you commit to anything. Your estimate should be against something which is clear and well documented. If things are not clear, it's quite possible what you deliver will not be adding much value.

Writedown the acceptance criterias

To have a clear scope of work it's a good idea to write acceptance criteria. Sometimes, certain tickets can have a lot of edge cases. A clear list of acceptance criteria brings clarity to the developer. And, it also helps the person who is allocating you the work plan better. You can refer to this article on how to write good acceptance criteria.

Break down big items into smaller logical chunks

Break down the stories or tickets into smaller chunks for estimation. Any ticket which is more than 3 to 4 days of work should be further divided into smaller ones. This not only allows more accuracy in estimate but also it shows continuous progress. Imagine you working on just one ticket for a week vs 4 smaller tickets completed in a week. Which one sounds better as progress? But yes, that doesn't mean you start breaking down the task without any logic or reason.

Discuss with your peer while estimating

It's always a good idea to discuss and take a second opinion during the analysis of the task before committing to an estimate. Sometimes, one might miss certain details / critical aspects of a task. Discuss with your team members about the task. It will help you doing better estimation. And, it will also mean others are aware of the functionality that you are working on.

Keep some buffer

As a developer, there are times when things don't work as you expect. To be honest, I feel surprised if my code works in the first attempt. And hence, it's a good idea to keep some buffer. If you feel you can develop a feature within 8 hours time, keep about 2 hours as buffer. This time should help you do some testing, code review etc. This will help you deliver better code quality.

Conclusion

As a developer, once you get better at estimates you will realise that you have acquired certain traits which help you mature as a developer. You will be able to work towards very specific targets and hence you will be able to plan your work better. You will get better at analysis of the task. A good developer is not the one who can just do bug free development but rather one who is able to develop the proper solution for a requirement.

Performance improvement & a silly mistake on my blog and how I fixed them

Amitav Roy — Mon, 04 Jan 2021 23:25:47 +0000

Recently I was working on performance improvement on my blog post. It all started when I was writing the article Mobile-first Google Indexing - Are you ready with your website for this change. During the Lighthouse scan one thing which was glaring to me was the size of the JavaScript file. And, that’s when I realized that I have made such a silly mistake. In this article, I am going to talk you through some of the important changes that I made to my blog. And, how they had a significant impact on the performance of my blog.

We all know performance is a very important thing when it comes to your website’s acceptance to the end user. Tons of research has confirmed that you get only a few seconds of the user before they move away from your website. I recently did a Page Speed audit of my website, and I was alarmed by the low score that I got for the mobile version. The desktop was around 91 which is great. But, the score on mobile was only 31. Can you imagine? The same URL on two different devices had such a huge difference. I had to do something about it.

Page speed and it’s recommendations

The Page speed report is a great starting point to work on the improvements. It’s always a good idea to look for the line items which are easy to fix and give you a good increase in the percentage. And, you know what I could see was one of the items in red was a JavaScript file size issue. It was a shocker for me as my blog has hardly any JavaScript behaviour. The only thing I recently added was the scroll indicator.

Digging a little deeper I understood the blunder that I had made. For getting the document ready event, I used the jQuery document ready event. And, that meant I was loading the entire jQuery library. And, that contributed to the huge file size. So, this was the first item which I had to fix. Upon Googling around, I could find a small NPM package “document ready” which would give me the same event. As I was very particular about the file size, I even went into the source code of the plugin and saw it will add a few lines of code only. This is why I decided to use it instead of writing my own.

After making this change, the app.js file size reduced from around 251 KB to less than 2 KB. And, this is why I said that I made a silly mistake. But anyways, the learning is that it’s very important to have a strong checklist while deploying your code to production.

Reducing first byte time using Redis Cache

After the file size improvement, I was looking for some way to reduce the first byte time. This is one of the most important aspects in reducing the waiting time for the user. This hardly took me an hour to implement across all the pages on my blog. But, it gave me a much better score on Page Speed and Web page test.

Redis is an in memory cache. To explain in simple terms, you are storing information in RAM of your server as a cached object. And, because RAM is much faster than hard disk IO speed, your data retrieval time reduces by a huge margin . I have a 2 GB server and I configured Redis to take up to 200 MB of RAM. Now, I know my entire website data won’t even take 20 to 30 MB. I have around 80 articles and that amount of text doesn’t eat up too much space right.

And, the implementation of the caching on Laravel is just a piece of cake. Whatever is your query result should be cached. And, before fetching the data from the database you just need to confirm whether you have that data in your cache or not. This means, now on page loads I am not at all hitting the database. The data is being served from in-memory cache which is way faster than establishing a connection to the database, query and getting the results. Below is a screenshot of the code which I have on the home page to fetch posts.

And the difference between non cached page vs cached page can be seen in the screenshot below:

Conclusion

So, from these above screenshots you can see that having a strong checklist is important. Along with that you should have a good tool set which you can use to get useful insights. With a quick deployment process, it is very easy to miss out on these aspects. Hence, a checklist is a good way to ensure that we are not making any big mistakes with the new code update (for example my jQuery file size issue). If I had checked the file size after the code update, I would have caught this.

Let me know what you think about having a checklist? Do you already have one, if yes how do you use it? You can get in touch with me through Twitter @amitavroy7 .

Banner image from Unsplash

Amazon (AWS) NLP Service - Comprehend and Twitter API

Amitav Roy — Sun, 21 Jun 2020 15:11:34 +0000

In today's world data is everywhere. And, so analysing data and coming up with insights from that data is a very important thing.

Recently, I was involved in a project where an automotive company wanted to get some insights about the sentiment of the users who were posting on their twitter handle and also on their other social mediums.

Now a days, using API we can fetch information from Facebook pages, Tweeter handles etc. And hence, it is possible to fetch that data, run them through Natural Language Processing (NLP) and get some idea about the content which is posted.

For example, as a social media manager I would like to know what is the sentiment of a user who has written anything about my company on Facebook or Twitter. If there is something positive written about my company, I would like to highlight that. If someone has written something negative, then we have an unhappy customer and I would like to know what was a problem and try to solve that problem.

Now, it is difficult to read through all these mediums and come up with analytics. And hence, services like AWS Comprehend can really help. Yes, there are a lot of services available who can do these stuff. But, I found AWS to be very easy, vast and cheap as well.

So, in this video I am going to show you how we can use the Amazon Comprehend service to fetch data from Twitter API using the Twitter streaming API and then send them to AWS Comprehend and get some sentiment analysis done on that tweet, extract some keywords and do other stuff.

So, if this requirement interests you, then dive into the journey where I show you how we can do this.

Setting up husky & lint-staged for code format and unit tests before git push

Amitav Roy — Sat, 30 May 2020 18:36:01 +0000

Having an automated code formatting and php unit runner is a great addition to your automation.

I work in a team where different developers have different habit when it comes to code formatting. Yes, they are small things I know. But, I spend a lot of time in code review. And, reading other's code is always difficult if the formatting is different. And hence, I want to ensure that the code formatting, structure etc are as much similar as possible. It reduces a lot of load from the code review process and speeds up things.

Plus, the aspect of automatically running the unit tests before every code commit is one check enforced for the developer to ensure that his new code is not breaking something which is running properly right now and was working before he / she made any code changes.

And, hence I looked at different options and this combination of husky to hook into the git hooks and lint-staged to get the context of the files which are staged are really helpful.

In this article, I will explain how I did the setup of these two npm modules to ensure code formatting and running php unit before every git push.

Hope you like it, and feel free to leave comments about your views.

https://www.amitavroy.com/justread/content/articles/setting-up-husky-lint-staged-for-code-format-and-unit-tests-before-git-push

Working with Laravel Livewire is fun

Amitav Roy — Sat, 09 May 2020 16:55:45 +0000

Development with Laravel is always fun. The more I spend time with Laravel, I enjoy the Test Driven Approach.

Now, when any frontend development comes into picture, there is an entire range of things to do. Browser testing, Unit tests, Mock servers to test your services and APIs and what not.

However, if you develop your frontend in Livewire, then even your front end code is in PHP. Which means, even your tests are in PHP. Plus, without writing any Javascript, you can create a complete SPA type application.

Under the hood, it doesn't return back JSON responses but rather HTML which is something even big giants like Github is also doing. And using smart dom diff, the new HTML is added to the dom and hence you get updates from the server based on user interactions.

It allows you to play will component lifecycle hooks as well which means, you can add behaviours based on change in state of the component and a lot more.

This in itself is a great thing. And, in this video I show you how you can get started with Livewire and start doing frontend development in PHP. Hope you like it.