DEV Community

loading...

How to build a URL Shortener like bitly or shorturl using Node.js

sachinsarawgi profile image Sachin Sarawgi Originally published at Medium on ・6 min read

In this blog, we will see how to build a URL Shortner like bitly or shorturl using NodeJS.

We may have heard many times that people are asking to build a URL shorter as an interview question, it’s not that complex but giving it a start to build one is complex though :).

So without wasting time let’s do it.

What is a URL Shortner

A URL shortener is a simple tool that takes a long URL and turns it into whatever URL you would like it to be.

Why we need it

Sometimes the links to a location or generally to a social platform become so big that it becomes difficult to manage them. A URL shorter will help in managing, track-compile click data, and one important point they promote sharing.

npm Packages We Are Going To Use

config : It lets you define a set of default parameters, and extend them for different deployment environments (development, QA, staging, production, etc.). For production, we have to define production.js similarly for development devlopment.js. By default, it will look for default.js.

This will be used to store config related to DB and others.

valid-url: This module collects common URI validation routines to make input validation, and maintaining easier and more readable. All functions return an untainted value if the test passes, and undefined if it fails.

This will be used to validate the URL given by the user for a shortening purpose.

shortid: ShortId creates amazingly short non-sequential URL-friendly unique ids.

This will be used to generate a unique id for each shortened URL.

express: The Express philosophy is to provide small, robust tooling for HTTP servers, making it a great solution for single-page applications, web sites, hybrids, or public HTTP APIs.

This will be used to create the server and route different HTTP path.

mongoose: Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment. Mongoose supports both promises and callbacks. As they use promise we will use async and await feature of JS.

This will be used for connecting with MongoDB, saving, updating and querying the DB.

Next, let’s setup MongoDB for setting our database.

Setting Up MongoDB Atlas

I wanted to use the cloud setup of MongoDB instead of a local setup, you can choose what fits better for you.

Steps for setting up cloud MongoDB Atlas account:

  • Go to Connect, create a user
  • Go to connect your application, you will see a URL (just remember the location of URL). Password will be replaced by your account password.

Setting Up The Project

Create a separate directory for your project urlshortner , open that directory in your favorite IDE. I am using Visual Studio Code here.

Go inside the folder and type npm init , give the necessary details for setting up the project.

Next, we need to download necessary node packages which we discussed earlier, type following command to download them

npm i express config mongoose shortid valid-url

The above command will update package.jsonwith the dependencies and download the needed packages inside node_modules folder.

Phewwwwww, Let’s Do The Coding Part Now

Open your code editor. Create a folder for storing the config, give the folder name config . Create a file inside folder default.js and give your MongoDB connect URL (we setup it earlier, I told you to remember it 😂) and baseURL.

  • replace the username and password with the user and password we created in MongoDB atlas.
  • allowedClick is a kind of restriction, that how many times the same URL can be used. Later can be used for pricing purpose. You can change it depending on your needs.

Config file for MongoDB setup

We will import necessary packages and connect with MongoDB

Define Schema For Storing URL Details

  • mongoose.Schema will define the document details which it will store. When we will code it will be much clear what each detail does.

urlCode : This will store the unique id related to each URL.

longURL : This is the URL which we need to short.

shortUrl : This is the actual short URL

clickCount : This stores how many times users have used the short URL.

Define Route for Shortening the URL

Create a folder name routes , inside that create a file shorturl.js which will have the code for shortening the URL.

Let’s understand the code.🤓🤓🤓

First of all, we imported the necessary packages which will be required later on. Used express package to create a route, using that route created an HTTP POST handler.

Next marked it async as it ensures that the function returns a promise, and wraps non-promises in it. Using async allow us to useawait , it makes JavaScript wait until that promise settles and returns its result.

We take out the URL submitted for shortening purposes from the request body, also fetch base URL which is mentioned in default.js . Next, we check if the URL submitted for shortening is a valid URL or not using the isUri method of valid-url package.

After the checking is successful we will query MongoDB to check if the URL sent for shortening already shorten or not. If shorten just return that result else short the URL.

mongoose methods returns a promise so we added await before it to wait till we get response.

For shortening the URL we will generate a unique id using generate method of shortid package. Next, append baseURL with the unique id to generate a URL as a short URL. Also, as the short URL is generated for the first time we will mark the clikcCount to be zero. Save the document and return the result as JSON.

Sample response (sending an amazon product links which need to be shortened).

Define Route for Redirecting the Short URL top Destination

Create a new file inside routes folder named getshortenurl.js .

Let’s understand the code. 🤓🤓🤓

First of all, we imported the necessary packages which will be required later on. Used express package to create a route, using that route created an HTTP GET handler. The URL will be getting shortUrl as a parameter. This parameter is the unique code that we appended to the baseUrl.

Next, we extract the shortUrl in a separate variable. As the code is unique so we can search the DB if we have any document with that unique code. The return result is stored in a variable.

If the return result is a document that means we have already shortened the URL. Check the clicked count of the returned document if the clicked count is passed the limit which we set in default.js, if yes return an error else increase the click count of the document and update it in the DB also, redirect to the long URL using the redirect method of res object.

Let’s Combine Everything

We need to have the main file which will combine all this together, remember we haven’t created the server yet. 😅😅😅

The baseURL which we configured in default.js has the value http://localhopst:8000/v1, because our app is running in localhost and the server is listening on POST 8000. The URL for getShortenUrlRoute is /v1/ so that is appended to the baseUrl..

Let’s understand the code.🤓🤓🤓

Import necessary packages with that import the routes which we have created in previous step, import config for MongoDB connection.

Connect to the DB, create the server and connect to a PORT (here it’s 8000).

app.use(express.json({})) this will parse incoming request body in JSON format.

Next, connect the router to the appropriate URL. Now let’s hope things work out. Start the app by using node index.js and play.

After creating a short URL paste the short URL in your browser it should redirect to the main URL.

I hope this blog will help you in understanding the basics of how to make a URL shorter. For code, you can refer to here on GitHub.

If you enjoyed reading this, don’t forget the like. 👏

Thank you.

If you enjoyed the content buy me a coffee. SachinSarawgi.

Discussion

pic
Editor guide
Collapse
eruizdechavez profile image
Erick Ruiz de Chavez

This is a nice example of a custom URL shortener. I myself have my own version in a php script (although completely different implementation).

I hope you don't mind if I leave some feedback here, as I think it would help you improve it.


Instead of using a config file, I suggest you to use environment variables. You can easily create default values, and override them on the fly with a start script, or something like dotenv.

In your index file you could do something like this:

const config =  {
    mongoURI: process.env.SHORTENER_MONGO_URI,
    baseURL: process.env.SHORTENER_BASE_URL,
    allowedClick: process.env.SHORTENER_ALLOWED_CLICK,
};

I think it would be better to validate the config below when the server starts, instead of waiting for a user to attempt to do something on the server to catch this error.

if(!validUrl.isUri(baseUrl)){
    return res.status(401).json("Internal error. Please come back later.");
}

Wrapping all your code inside an if is harder to read. It is considered a good practice to fail fast so you avoid unnecessary nesting.

Instead of this:

 if(validUrl.isUri(longUrl)){
// …
else{
    res.status(400).json("Invalid URL. Please enter a vlaid url for shortening.");
} 

You could do this:

if(!validUrl.isUri(longUrl)){
    return res.status(400).json("Invalid URL. Please enter a vlaid url for shortening.");
} 
// …

Users do not need to know your internal implementation details. Hide them by just returning what they need on their response:

Instead of returning this:

{
    "_id": "5eec662da11d145bd0be75da",
    "longUrl": "https://www.amazon.in/gp/product/B085VTLD43?pf_rd_r=MV8SV74D900FNTWRWSMT&pf_rd_p=649eac15-05ce-45c0-86ac-3e413b8ba3d4",
    "shortUrl": "http://localhost:8000/v1/bxd5-mqVB",
    "urlCode": "bxd5-mqVB",
    "clickCount": 0,
    "__v": 0
}

You could return this:

{
    "longUrl": "https://www.amazon.in/gp/product/B085VTLD43?pf_rd_r=MV8SV74D900FNTWRWSMT&pf_rd_p=649eac15-05ce-45c0-86ac-3e413b8ba3d4",
    "shortUrl": "http://localhost:8000/v1/bxd5-mqVB",
    "urlCode": "bxd5-mqVB",
    "clickCount": 0,
}

In your get handler, the user does not need to wait for the click count update to be redirected. You should respond to the user as soon as you can and let node do anything else after the user is well gone.

Instead of this:

await url.update({ clickCount });
return res.redirect(url.longUrl);

You could do this:

res.redirect(url.longUrl);
return url.update({ clickCount });

Last but not least, if you want to further optimize this specific example, you could remove Express and use Node's builtin http server; that would reduce a lot or dependencies you are not using at all. Another good option is to use restify which is pretty similar to Express but with less dependencies and focused on API development and not full fledged web apps.

Collapse
sachinsarawgi profile image
Sachin Sarawgi Author

Hi Erick that's a lot of valuable suggestions....I didn't know how to do env var thing in NodeJS.....but now I am surely going to use it.
Thanks for the guidance.

Collapse
eruizdechavez profile image
Erick Ruiz de Chavez

I use it a lot and it is very portable, specially when deploying your code to Docker images, or cloud services. I also use dotenv for my local development and I do not even need to include it in my source code; I just install dotenv as a devDependency and then start my local server with node -r dotenv/config myfile.js, that -r automatically requires your module.

github.com/motdotla/dotenv#preload

Thread Thread
sachinsarawgi profile image
Sachin Sarawgi Author

Thanks for mentioning the required steps.... I will update the code

Collapse
karimerrahli profile image
karim

Great guide!
Next step could be implementing a caching system like redis to store latest frequented URLs which would save db access.

Collapse
sachinsarawgi profile image
Sachin Sarawgi Author

Thanks karim for the idea....I will give it a try when I write next regarding this.