DEV Community: Shobi

An In Depth guide on deploying fastapi to AWS ec2

Shobi — Thu, 02 Feb 2023 18:40:35 +0000

In this video, I would like to show you in-depth details on how to deploy a python Fastapi app to an AWS EC2 with Nginx, supervisor and uvicorn. Also I have added timeline marks to easily navigate through the video

The configuration files and sample application can be found in the github repo: https://github.com/ShobiExplains/fast...

The readme also contains some commands that you can use. https://github.com/ShobiExplains/fast...

Follow me on twitter : https://twitter.com/ppshobi

Please post your comments, doubts and Please subscribe and share the video.

Database replication as a scalability tactic in Relational Databases

Shobi — Mon, 16 Jan 2023 22:57:00 +0000

Intro to Database Replication on Relational Databases

I would like to call Replication the twin brother of Duplication. Even though these words are interchangeable most of the time they can't be interchanged all the time. Let's look at an excerpt from this Stack Exchange answer

If I'm a good painter, I can replicate the Mona Lisa, I cannot 'duplicate' it! So, to answer this question, check twice before interchanging these words.

Likewise, when you lose your original driving license or any other document, you apply for a duplicate copy. There, 'replica' does not work!

When your App starts to have performance issues with an increase in the number of requests, database replication is one quick way to breathe a little bit of performance into your app. a Database replica is a duplicate copy of your database. But it's replicated meaning the data is reproduced in the replica the same way it was initially created in the original database.

In this write-up, I would like to focus more on the replication technique of Relational databases like MySql, Postgres, etc. Also called a Read Replica, as the name suggests the replicas are read-only. Just because I can simply touch the idea of replication alone and don't have to write about Clusters, Leadership, Quorum, Election, etc... but could be a possibility in another post.

Application of Database Replication

Database replicas are an excellent way of offloading a heavy number of requests from a database. A replica is another database instance/server of usually the same configuration (at least the storage capacity) let us look into a simple Single Master Single Slave setup for a database and see how it functions

The main database called Master has a replica called Replica 1, The user is interacting through the App with the database, and it issues quite a number of database queries. All the write queries are going to the Master database initially and whatever data is inserted/updated/deleted will be replicated to the Replica 1 database Eventually. Later on, every read query is made to the Replica. This kind of configuration is one of the simplest and really helps to increase performance in a ready-heavy application by routing the read queries to another database and freeing up the Master database.

This replication configuration could be extended further by adding more replicas and forming a group of replicas. see the following figure

In this case, the write that happened in the master database is replicated into multiple replicas. The read queries are made to any of the replicas and the replica will respond with whatever latest data it has. We can use replicas to achieve High Availability and by deploying the replicas in multiple regions to avail Low Latency.

Usually, the replicas are used/consumed by the App by keeping different database connections to the Master and Replica(s). During a user request, the App chooses which connection to use, whether the Master connection is to do a write operation or a Replica connection to do a read operation. Another excellent choice would be to use replicas as the source of the Business Intelligence tool. You don't want heavy aggregations/operations being done to your master DB when you have users.

How replication is done

The replication could be done in two ways

Synchronous
Asynchronous

In synchronous replication, the database finishes the transaction only when both the Master and Slave(s) have the data. While this ensures the data is available/consistent across master and slave databases. Obviously, synchronous replication greatly reduces the write performance, especially when the replicas are not in the same data centre/region eg: Multi-AZ replicas.

While synchronous replication could be required in specific scenarios Asynchronous replication is my preferred way and it's the default for many cloud vendors including AWS. In asynchronous replication, the master completes the write and the changes are propagated to other replicas eventually. This is also a great weakness of replication and you can see that in the cons section.

The exact details of replication depend on the specific databases. But generally, the master DB produces the events for the operations that happened and these events (logs or binary data) are copied and replicated in the replicas, another way is to look for the primary keys in the tables and incrementally copy them.

Support for replication

Most relational databases have replication support.

Mysql
Postgres
Microsoft SQL Server
Oracle etc... are to name a few.

Major cloud providers also have managed replication solutions

Just a few clicks and you have a replication.

Pros of Database Replication

Availability - Even when the Master DB is failed, the replica can still serve the users in a read-only mode.
Disaster Recovery - In a situation of master DB failure, the replica can be promoted to a master fairly quickly and keep resuming the service. Most of the vendors have this failover support (Sometimes not fully automatic). Replicas can also act as a fairly decent backup.
Performance - Database replication in theory could make your application perform very well in typical use cases.
Simple - Replication is not a super complex topic to understand and is fairly quick to implement. While manual replication setup is not that simple, with modern cloud providers it is a matter of a few clicks to have the replication setup. Also, you can read a bit more about manual setup here. There are many 3rd party tools and providers that focus on replication technologies.

Cons of Database Replication

Database replication can be a quick way to add some performance to your apps and will serve you to great lengths in terms of performance. But it is not a silver bullet that solves all the problems, hence let us look at some limitations of database replication.

Consistency - It is one of the biggest weaknesses of the replication. The chances of inconsistent data are very much of a problem. When the master has a write operation and for some reason, the operation is not propagated, the replica will respond with inconsistent data forever. To avoid this some kind of data integrity checks need to be deployed. And this is not really a trivial task. When an inconsistency is detected, fixing it would also not be trivial, especially in a big database with many foreign keys and stuff. Usually fixing inconsistency involves building a new replica and throwing the inconsistent replica away.
Latency - When the user would not see his updated data in a mission-critical solution it would not take much time for him to look for another app. This is especially common in the case of multi-region replicas. For a user, his trivial activities are mission-critical and we want to show him his updates instantly. So it's important to monitor replication time and verify it's within acceptable limits. For example, I would use a master DB connection to manage his current order and probably will use a replica to pull the order history.
Cost - This is something obvious. The replicas are their own new instances. You will pay for CPU, RAM and Storage. Even though you can have a smaller replica instance in terms of CPU or RAM, still storage needs to be the same as the original database.

Like I mentioned DB replication is not a solution for all your database problems, but it can be a wonderful addition to your toolbelt, and replication is supported by most of the mainstream database and cloud vendors. Let me know what you think about replication in Twitter @ppshobi

originally written https://shobi.dev/blog/2023-01-14-database-replication

I have created a video series to help you deploy a Laravel application with PHP7 and Nginx

Shobi — Fri, 27 Mar 2020 03:29:29 +0000

Hello, after a long time I had enough time to edit my recorded screencasts on deploying a laravel application. The corona time helped me, a lot of time is saved on getting ready for office, commuting, etc...

In this tutorial video series, I will show you how to deploy a laravel application with PHP7.4, if you are using any PHP7 version it should be fine as well. As of now, PHP7.4 is the latest version so I have used it as well.

For the demo purpose, I have used a fresh installation of laravel without any database connectivity, but once you have deployed application this should be fairly simple for you to do, given how easy to set up a managed database these days. But still, I have plans to show you how to install and configure databases in another video.

Below are the links to those videos, feel free to subscribe and post your doubts, comments and suggestions. also don't forget to subscribe if you find it useful.

Part 1 - Laravel Deployment in Depth - Provisioning Server and Installing Nginx

Part 1 Extra (for aws users): Provisioning server in AWS

Part 2: Creating a custom user to secure the server

Part 3: Installing PHP & Connect to a custom domain

Part 4: Deploying the application to the server

Github repo of the demo application

ShobiExplains / LaravelDeployment

I wrote a PHP Client for sonic, the autosuggestion engine, and now its official PHP client. Ask Me Anything 🤩

Shobi — Thu, 11 Apr 2019 18:23:58 +0000

Hey guys, last week I got to see one article about a new search engine, similar to elastic search. but built specifically for autosuggestions. Sonic - it is an index based auto-suggestion engine built on Rust lang by the team at crip.chat . a customer engagement platform.

Then I went to their repository (Yep, its open source). And found out that it offers a TCP endpoint rather than an HTTP endpoint/. Meaning in order to use sonic search, you need to connect to the TCP socket and issue commands over the socket and read the output buffer. They are officially offering a javascript client for sonic but not for PHP. Then it struck me... This can be a good experience for me if I build a PHP client for it, especially because I haven't done any socket programming before.

Then I spent a few days and nights cooking the PHP client and submitted a PR into the official repo,

Added one more php client - psonic by ppshobi #118

ppshobi posted on Apr 06, 2019

Hi, I have gone ahead and implemented a PHP client for sonic, including unit tests and a relatively complete readme and an api documentation as well.

View on GitHub

Saying that kindly include my client as well in the readme. Then I tweeted about the same in twitter as well.

// Detect dark theme var iframe = document.getElementById('tweet-1114835655450718208-884'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1114835655450718208&theme=dark" }

Ahaa... There it was. The CTO of crisp and the core contributor of Sonic, Mr. Valerian Saliou replied to me saying that, he liked the way I wrote the code and he can promote my client as an officially supported sonic client for PHP if I come up with a relatively complete solution.

Well then..., few more nights and voila. Now Psonic is the official php client for sonic. Check it out, and give your valuable opinions in the comments as well as GitHub stars 😍.

ppshobi / psonic

Sonic is a super fast auto suggestion engine built by the team at crisp.chat, a customer engagement platform. its built in Rust and they officially support a javascript client, but if you want to use sonic via php, this is the library that you can look for. Completely unit tested, and adheres to modern coding standards, and offers a clean api to interact with sonic.

Psonic - PHP client for sonic auto suggestion engine

Summary

Sonic is a super fast auto-suggestion engine built by the team at crisp.chat, a customer engagement platform. it is built in Rust and they officially support a javascript client, but if you want to use sonic via PHP, this is the library that you can look for Completely unit tested, and adheres to modern coding standards, and offers a clean API to interact with sonic.

Installation & Usage

you need a running sonic instance (locally or in the cloud, the port 1491 should be accessible) php7+ and composer to make this library work. Read more on installing sonic

goto your project directory
execute composer require ppshobi/psonic once the installation is completed you should be able to use the library as follows

Api Documentaion

Full API Documentaion

Usage

once you have psonic in place, you have access to the Client…

View on GitHub

Go ahead and try sonic as well. It's a really cool product. 😍😍😍

Part 2- An in-depth video tutorial on deploying a Django application with static file serving

Shobi — Fri, 22 Mar 2019 07:17:38 +0000

Django is the most popular Python-based web framework for a while now. Django is powerful, robust, full of capabilities and surrounded by a supportive community.

In this video, we are going to take a look into how to set up static file serving in our app, such as CSS, javascript, images etc... in our simple Django application. which we built and deployed in Part 1 of the series

Static files on Django should be served by nginx directly rather than our Django app. So you need to add a few lines to your nginx configuration, to tell where your static files are. I have tried to explain things neatly in this video. feel free to as any doubts.

Let me know the suggestions, thoughts & the doubts in the comments. Subscribe to the channel & Share this with your friends & colleagues

Link to sample application repo with static files already setup: https://github.com/ShobiExplains/AwsDemo/releases/tag/v0.0.2

Nginx Configuration for reference: https://github.com/ShobiExplains/AwsDemo/blob/master/django.conf

An in-depth tutorial on deploying a Django application into AWS

Shobi — Fri, 01 Feb 2019 11:24:42 +0000

Django is the most popular Python-based web framework for a while now. Django is
powerful, robust, full of capabilities and surrounded by a supportive community.

In this video, we are going to take a look into deploying a simple Django application to an AWS ubuntu instance, with Nginx, Gunicorn & Supervisor configuration.

In this video, I have tried to explain every single step & each line of code I have written. Hopefully, this video will help you out.

Let me know the suggestions, thoughts & the doubts in the comments. Subscribe to the channel & Share this with your friends & colleagues

Shameless plug: I am building a tool to automate server provisioning and deployments for django and other python apps. The free early access is availble at pyployer.com

Link to sample application repo: https://github.com/ShobiExplains/AwsDemo

How do I tie Laravel and Lumen together?

Shobi — Tue, 30 Oct 2018 06:58:16 +0000

I have asked this question in SO, But no responses after 20 Hours, if any of you guys can help it by either answering here, or give an upvote in SO so that it gets more attension.Original Stack overflow question

I am building an application from scratch which has 2 main parts

A Laravel server - Which will serve HTML pages by querying an API (Preferably no DB interaction)
A Lumen Server - Which will be the API being queried (DB interaction goes here)

Now I need to log in a user.

I can send username and password from laravel server to the lumen API and get a token back (I think it's the way, correct me if I am wrong).

After that what should I do to make the user logged in IN the Laravel server?
Keep the token in session with some key?
How do I make this work with the Auth::user() functionality in the Laravel Server?
Or what are some good approaches to this problem?

I have created a chrome extension to paste random data to form fields 🎷🥁

Shobi — Thu, 18 Oct 2018 12:16:59 +0000

Hey guys,

Since the last two days I was working on a chrome extension, which can insert fake data to form fields. and finally I have published it in chrome web store, you can check it out ⚡️ here ⚡️

I was tinkering around typescript for a few days and then I thought yes. I should do something with typescript.🤠

And one day my colleague was testing a feature where he needs to fill a lot of forms, he was copying lorem ipsum from one site and pasting it into the form field. I told him, Man... you can do better, check for a chrome extension and you can fill the form easily for testing purpose.

He did a search and found some extensions which were helping him. but still, the copy paste part was involved, (ie: you click on the extension icon, it will give a popup and you need to copy it from there and paste it in the form. ) which of course was not up to the mark.

And moreover there is no support for data like a credit card, names, address, zip code etc... and yes ☃️, I got my idea for a chrome extension and again I can try doing it in typescript and it will be good to learn the chrome API's as well. yay... yay...

I took faker.js as a starting point and it can generate almost any type/amount of fake data as required.and its a widely accepted library for unit testing purpose as well.

But as the project progressed, it took 2 entire nights to figure out the typescript intricacies, and the chrome extension architecture, Then I was like, okay... rather than doing a full-blown extension, let me create a version, where it can give a random text for a given length and publish a beta. Then from there I can keep adding features.

Today I did it 🍾🍾🍾

So at this point, after the installation of the extension, you need to press
Alt + Shift + I, and then give a length in the prompt box. Your last focused input field will be filled with random data.

Extension Url - https://chrome.google.com/webstore/detail/random-text-generator/lmealcdbihjhocippaajjhijhjciaojo
Github Repo - https://github.com/ppshobi/chrome-random-text-generator %}

So guys... please test it out. Let me know the issues here or in the github issues page. If you would like to contribute I would be glad to guide you as well. ❤️❤️❤️

Short Demo Video

Email Sending in Django 2, Part -1

Shobi — Wed, 10 Oct 2018 07:02:04 +0000

Yay ... I made another django video,

Emails are an integral part of any web application. You need them for user verification, billing or any kind of communication with users.

In this video, we are taking a look into django's console email backend, We will be sending emails as usual, but instead of actual email, Django will display it in the terminal. Let me know what you guys think in comments here or in youtube, I am happy to help you resolve any issues you encounter.

Source code : https://github.com/ShobiExplains/SimpleDjangoAuthentication/releases/tag/v0.0.2

I have created a tutorial video on how to build a login system in Django by leveraging Django's built-in auth component

Shobi — Wed, 03 Oct 2018 08:17:51 +0000

At some point for a web application, we need the authentication feature to identify the user and manage them.

Being a battery included framework Django comes with a builtin login system. So I have created a video screencast with step by step instructions to build an auth system in Django.

In this screencast, we will be taking a look into how to leverage Django's power to build a powerful authentication system

Here is the link http://bit.ly/2NXLgnf

Source Code : https://github.com/ShobiExplains/SimpleDjangoAuthentication

Let me know what you guys thinks about the video.

A look into Laravel and its Ecosystem

Shobi — Wed, 19 Sep 2018 20:37:25 +0000

As a developer working primarily on PHP and Laravel I have come across a multitude of services and products related to Laravel. These things made Laravel a popular (Extremely popular I would say) among PHP developers.

Even though criticisms are present against Laravel being batteries included framework, its popularity skyrocketed since its stable release 4.2 launches in June 2014.

See the Google trends chart for past 5 years

I would say Laravel gained its momentum not only because its a developer friendly framework or a robust framework; rather because of its surrounding ecosystem of products and services which would help in crafting an application faster. Much like Apple's products and ecosystem.

As of August 2018, Laravel 5.7 is released and with the ecosystem, it's a whole lot powerful. Let me list out some of those here. Even though I haven’t used all of them in production, I would consider these things as important as laravel itself.

1 . Laracasts.com

Robustness of a frameworks doesn’t help when there are not enough resources to introduce or teach them to the developers. At least in my case, I am super lazy to learn a new framework just because the learning that I get from it is nothing but how the framework does a specific thing? How routing is done in Framework X doesn’t make any change in what routing is.

Laracasts is one of the obvious names in the laravel world. Jefrrey Way produces upto date screencasts on laravel and related technologies. I personally found the videos quite entertaining and informative (Its the Netflix for developers). Laracasts is not only about Laravel but as an integral part PHP and its intricacies are introduced.

2 . Laravel.com

No matter what the software is, it's incomplete and unusable without its documentation. Laravel.com contains the documentation for the framework, officially supported packages such as Scout, Passport etc… and related subjects such as setting up a development environment, deployment instructions etc…

Being an open source project the laravel documentation is constantly updated and accurate as well.

3 . Laravel Homestead & Valet

Setting up a development environment as similar as to the production was a pain in the ass previously, but with vagrant and virtualisation, this task is as simple as installing few softwares, and no need to worry about cross platform issues as well.

I feel pity for those who use xampp or mamp and whine after deployments. Time to switch to the new world dudes…

4 . Envoyer

Envoyer is a deployment tool built on top of laravel itself. It can hook into your version control systems events, and trigger deployments automatically. Envoyer can deploy any PHP application on AWS, Digital Ocean etc… (Of course tuned for Laravel as they say).

5 . Laravel Forge

Laravel forge is a server provisioning and managing service. It can create and manage PHP servers for you on AWS, Digital Ocean or Linode. It can provision SSL certificates and can manage deployments as well.

Forge gives you cutting edge Linux server configurations and also supports horizontal scaling as well.

6 . Laravel Lumen

Lumen is the small brother of Laravel. A slimmed down version of Laravel with the primary focus on the development of api's and micro services.

A goto software for building API based apps.

7 . Laravel Spark

I wouldn’t say spark is an integral part of laravel developer ecosystem. But to scaffold an application super fast you have laravel spark. Spark can help you to start an application with ready-made Billing, Subscription, Invoices, Notifications etc…

8 . Laravel Nova

Nova is a recent addition to the ecosystem. Laravel Nova provides you an admin panel for your application. With the addition of ready to use CRUD components, you can build an admin section super easy.

Typically nova doesn’t need any database tables to manage itself. It's completely code driven. You don’t have to touch the existing laravel models at all.

9 . Laravel Horizon and Queues

Laravel ships with queue component out of the box. You can view and manage your asynchronous jobs and queues with the horizon in a breeze.

Horizon integrates with Redis and it can retry failed jobs automatically based on the specified configuration which makes our life a whole easier.

10 . Laravel Shift

Laravel Shift is a service which can automatically upgrade Laravel versions on your behalf. All you have to do is give access to your git repository and it will upgrade your codebase by creating a detailed pull request.

11 . Laravel News
Since 2012 when the Laravel initial release came out, Laravel news has been bringing the latest news, tutorials and podcasts to us. I would recommend everyone to have a look there to get the latest updates on Laravel.

Final Notes

PHP runs the ~70% of the web. Being an opensource project with ~46k stars in GitHub as of now, Laravel really stands out. It means nothing but its going to be around for a while. With official packages such as Cashier for billing, Scout for instant search etc... it's even more powerful and almost every day new features and bugfixes are being added to the framework.

🔥🔥 I would say stay tuned for more excitement 🔥🔥

Laravel Collections Higher Order Messages - Single Line Magic

Shobi — Fri, 17 Aug 2018 21:11:38 +0000

Collections are wrappers around arrays in PHP. Collections provide fluent interfaces and intuitive APIs to access arrays. Laravel comes with collections out of the box, and Laravel's eloquent ORM returns the result set in the form of collections by default.

Collections come with many useful methods which allow us to easily manipulate arrays, and they are chainable as well. Compared to the built-in PHP array manipulation functions, collections offers much readable and consistent syntax.

Here I will show you a little refactoring technique you can do while tinkering with collections using higher order messages.

Collections provide support for "higher order messages", which are short-cuts for performing common actions on collections. Documentation

Consider the scenario where you need to map over a collection and call method on the item

<?php
User.php
--------- 

/** 
 * the method returns the full name of the user 
 * by joining first and last name 
**/

function fullName(){
   return $this->first_name . ' ' . $this->last_name;
}

SomeFile.php
------------
/** 
 * In some other file, we need full names of all users 
**/

$fullNames = User::all()->map(function ($user) {
    return $user->fullName()
});

the above code shows how we do it normally with map(), but functions with the simple body as in the above code, we can use higher-order functions to cut the explicit crap.

<?php
$fullNames = User::all()->map->fullName();

Boom ⚡ Single line Magic 🔥❤️

Let's goto a little more complex scenario where we need to filter over a collection and then map over them again

<?php

User.php
---------
/** 
 * the method returns the full name of the user by joining first and last name 
**/

public function fullName() {
   return $this->first_name . ' ' . $this->last_name;
}

/** 
 * the method returns the if the first_name starts with a given letter 
**/

public function isStartingWith($letter) {
    return starts_with($this->first_name, $letter)
}

OtherFile.php
-------------
/** 
 * In some file, we need full names of all users which starts with 'A'
**/

$fullNamesWithA = User::all()->filter(function($user){
    return $user->isStartingWith('A');
})->map(function ($user) {
    return $user->fullName()
});

Let's try to use Collection Higher-order functions here and cut explicit syntax crap.

<?php
$fullNamesWithA = User::all()->filter->isStartingWith('A')->map->fullName();

Boom ⚡ Single line Magic 🔥❤️ Again

Adiós